Designing agentic systems that can be trusted

A suitable starting point is a repetitive, high-volume workflow with a clear outcome. Consider supplier invoice handling: an inbox receives invoices in a dozen formats, someone matches each against a purchase order, resolves discrepancies by email, and posts the result to the finance system. Before choosing a model or framework, we map every decision in that flow, the data each decision needs, the systems it touches, and where responsibility passes between people and software.

That map defines the agent's boundary. Extracting fields from a PDF and matching them to a purchase order is well suited to a model. Deciding whether a $40,000 discrepancy is a fraud signal or a known contract variation is not, at least initially. That step needs a human decision, a deterministic rule, or a conventional software component. The design should state which steps are delegated, checked, or prohibited.

Mapping first also prevents a general assistant from being built without a defined use case. An agent designed for a specific workflow inherits that workflow's definition of success, which makes quality, safety, and value easier to evaluate.

Tools define much of an agent's capability and authority. A raw database connection gives the model more access than this workflow requires. A tool named match_invoice_to_purchase_order can instead accept an invoice identifier and return a structured comparison. Narrow, well-described tools reduce the available error paths and make each call correspond to a business action that a person can review.

Permissions belong to tools, not to the agent as a whole. The invoice agent might read from the finance system freely, write draft journal entries that require approval, and have no path at all to payment execution. Structuring authority this way means the worst case of a confused model is a wrong draft, not a wrong payment.

Tool design also controls how an agent interacts with other systems. Side-effecting calls need idempotency keys. Timeouts and retries need bounded backoff, and errors need a structure the model can interpret. These controls determine whether an agent can recover predictably from failures.

A production agent needs an operational trace for every run: the inputs it received, the tools and arguments it used, relevant evidence and decision summaries, and the outcome. Private model reasoning is not required and may not be available. An operator investigating a held invoice or approved mismatch should be able to reconstruct the system's observable decisions.

Observability supports systematic improvement. A miscategorised invoice becomes an evaluation case; a cluster of similar failures prompts a review of the instructions or tools; repeated misuse of a tool indicates that it may need redesign. Traces let the team identify recurring failures and decide whether the evidence supports a wider operating scope.

Models produce incorrect results, and the design must specify the system response. Routing thresholds can send uncertain or high-risk cases to a person. Approval gates can prevent consequential actions from proceeding automatically. Rate and value limits constrain the potential damage from repeated errors, and an operator must have a direct stop control. Reversible actions also need a defined rollback procedure; for example, a draft entry should have a documented removal or correction path.

Define failure handling before writing the system prompt. For each tool, identify the worst plausible misuse and the control that would contain it. If no control contains that misuse, the tool grants more authority than the current design can support.

A staged release limits direct execution risk. The agent first runs in shadow mode, producing recommendations that people review while the team builds an evaluation set from real traffic. Shadow mode still carries privacy, data-handling, and review risks. The agent can then handle a low-consequence segment, such as invoices under a threshold with exact purchase order matches, while other cases route to a person. Scope widens only when evaluation results and observed failures support the change.

The people accountable for the workflow approve each increase in scope against defined evidence. This staged process supports unsupervised operation only for tasks that have passed the applicable quality and safety checks.