An agent differs from an automation in exactly one way that matters: it decides which actions to take. That is what makes it useful, and it is why an agent with production permissions is a security design problem before it is an AI problem.
Most agent demonstrations skip the interesting part. They show a model calling tools successfully. They do not show what happens when it calls the wrong tool, calls the right tool with wrong arguments, or is manipulated by something it read into doing neither.
We build agents the way we would build any system with production access. Least privilege, narrow capabilities, everything logged, and a human in the loop for anything expensive to undo.
Start from the blast radius, not the capability
The question that shapes the design is never “can the agent do this.” It is “what happens the time it does this wrongly.”
That reframing produces different architecture. It moves the important work from prompt engineering to permission design, where it belongs, because prompts are guidance and permissions are enforcement.
A useful test: for every tool you are about to expose, write down what a single mistaken call costs and how you would reverse it. If you cannot answer the second part, the tool needs an approval gate rather than a better prompt.
Narrow tools beat broad ones
The most common serious mistake we see is a single powerful tool where several small ones belong.
# Do not do this
def run_sql(query: str) -> list:
"""Run a SQL query against the production database."""
return db.execute(query)This tool has the full permission of whatever database user is behind it. A mistaken call is unbounded. It is also, incidentally, harder for a model to use correctly, because it has to construct valid SQL rather than supply two typed arguments.
# Do this instead
def update_order_status(order_id: int, status: Literal["packed", "shipped", "cancelled"]) -> dict:
"""Set the fulfilment status of a single order.
Only these three statuses. Cannot modify payment state, customer records or
any other order field.
"""
...Narrow tools give you three things at once: a bounded failure, an argument set you can validate, and
an audit log entry that means something. update_order_status(order_id=8412, status="shipped") is a
readable record. run_sql("UPDATE ...") is an archaeology exercise.
Separate read from write while you are at it. An agent that can read broadly and write narrowly is both more useful and far safer than one with symmetric access.
Treat everything the agent reads as untrusted
This is the failure mode that is genuinely new, and the one most teams have not internalized.
If your agent reads an email, a support ticket, a web page, a PDF or a code comment, that content can contain text addressed to the agent. OWASP catalogues prompt injection as the top risk for LLM applications for good reason: it turns any content your agent processes into a potential command channel.
The naive mental model is that instructions come from you and data comes from elsewhere. The system does not inherently distinguish them, so you have to.
What does not work reliably:
- Telling the model to ignore instructions found in content. It helps, and it is guidance, not a control.
- Filtering for suspicious phrases. The space of phrasings is unbounded.
- Assuming your content sources are trustworthy. A supplier’s invoice PDF is not under your control.
What does work:
Architectural separation, so that a successful injection still cannot do much.
- Keep the tool-calling agent away from raw untrusted content where possible. Have one component summarize or extract structured fields, and pass only those fields to the component with tools.
- Gate every consequential action behind human approval, so injection can propose but not execute.
- Scope credentials to the task rather than to the user, so a compromised agent cannot reach beyond its purpose.
- Log the full context, so if something odd happens you can see what the agent read immediately before it happened.
The last point matters more than it sounds. In the incidents we have looked at, the transcript immediately preceding a strange action is almost always where the explanation is.
Approval gates that people actually read
An approval step that always looks the same gets approved reflexively within a week. That is not a control, it is a delay.
A gate is only meaningful if it shows the specific consequence:
Bad: "The agent wants to perform an action. Approve?"
Good: "Cancel order #8412 (Sarah Ahmed, AED 2,340, placed 3 days ago,
currently marked packed).
This will trigger a refund and notify the customer.
[Approve] [Reject]"The second one gets read because it contains information the reviewer can evaluate. The first trains people to click.
Reserve gates for things that deserve them. If everything requires approval, nothing does, and users will look for a way around the whole system.
Idempotency, because retries happen
Agents retry. Networks fail mid-call, models produce a malformed response and try again, an operator re-runs a job.
Without idempotency, a retry means a second refund, a duplicate order, two emails to a customer.
def issue_refund(order_id: int, amount_cents: int, idempotency_key: str) -> dict:
"""Issue a refund. Calling twice with the same idempotency_key is a no-op."""
...Derive the key from the intent rather than from the attempt, so a retry of the same logical action carries the same key.
Log the whole decision, not the outcome
The question you will be asked is “why did it do that.” Answering it requires more than a record of what happened.
For each action, retain:
- The trigger, and who or what initiated the run
- The context the agent had available at that point
- The tool called, with its exact arguments
- The result returned
- Whether an approval was requested, who responded, and what they saw
That last item is the one people omit and later need. “The user approved it” is not a defence if nobody can reconstruct what the approval screen said.
Give the agent its own identity rather than borrowing a person’s credentials. If agent actions are indistinguishable from that person’s in the audit log, you have lost attribution and you cannot revoke the agent without revoking them.
Earn permissions with evidence
The pattern that works: start read-only, run supervised, read the transcripts, and widen scope because the logs justify it.
An agent that gathers context, drafts a reply and recommends an action is useful on day one and carries almost no risk. Adding the ability to send that reply is a separate decision, made after you have seen a few hundred drafts.
Real transcripts surface failure modes no specification anticipates. In our experience the surprises are rarely dramatic: the agent handles an ambiguous case confidently rather than escalating, or misreads a piece of formatting, or takes a reasonable action based on a stale record. None of that appears in a demo, and all of it appears within a week of real traffic.
A stop mechanism you have actually tested
Every agent needs an off switch, and the off switch needs to have been used at least once.
- Something a human can trigger without a deployment
- A rate limit, so a loop cannot execute a thousand actions before anyone notices
- A budget ceiling, for the same reason in a different currency
- An automatic halt on an anomalous pattern, such as the same tool called far more often than usual
Test it under load rather than assuming it works. A stop button that has never been pressed is a hypothesis.
Where the accountability sits
Worth stating plainly, because it gets left vague: the organization deploying the agent is accountable for what it does. Not the model provider, not the framework, not the engineer who wrote the tool definitions.
That is why the design questions above are business questions rather than implementation details. Which actions require a human is a decision about acceptable risk, and it belongs with whoever owns that risk.
Our job is to make sure mistakes are bounded, visible and reversible. Narrow tools bound them. Logging makes them visible. Approval gates and idempotent, reversible operations make them recoverable. None of those make the agent correct, and all of them make being wrong survivable.
When not to build an agent at all
If the sequence of steps is fixed and no judgement is involved, an agent is the wrong tool. A deterministic automation is cheaper, faster, more reliable and vastly easier to reason about.
Agents earn their complexity when the next step genuinely depends on what was found. That is a narrower set of problems than the current enthusiasm suggests, and being honest about which one you have is the first design decision.