Agent Backup System: recoverable state for autonomous agents

This reference design separates recoverable state into four layers captured at the same logical checkpoint. The first is working memory: the transcript, explicit plan, intermediate results, and temporary variables available to the runtime. Private model reasoning is not a required backup component and may not be available from the model provider. The second is episodic memory: the durable record of past interactions, learned rules, and user preferences. It typically resides in a vector store or structured memory database that must be synchronised or referenced consistently with the checkpoint.

The third layer is execution state, represented as a task tree with a cursor. If the agent has a ten-step plan, the snapshot records completed, active, and pending steps. It also records deferred work and the event cursor needed to resume an append-only journal. The fourth layer is environment and tool state: which tool handles existed, what the browser was showing, which operations were active, and which credentials the agent could use. The snapshot records references rather than credential values. Many connections and tool sessions cannot survive a runtime failure, so the restore process recreates or reconciles them instead of assuming that they can resume.

Each layer is described in a signed manifest with an agent identity and schema version, rather than hidden inside an opaque runtime image. Secrets never enter the portable state: the manifest stores references to credentials and external resources, and the restore environment resolves them through its own secret manager and access policy. Model weights, container images, and large immutable artefacts can likewise remain external when a content digest and a durable retrieval location are sufficient.

This separation keeps a backup useful across runtimes. It also makes the recovery boundary visible: the service preserves agent state, not every dependency or external system the agent has ever touched.

An agent can invoke the service through a custom Model Context Protocol tool such as create_backup or through an authenticated API endpoint such as POST /v1/backups. MCP provides the tool invocation mechanism, not the backup format or durability guarantees. The request identifies the agent, state schema, checkpoint mode, retention policy, and an idempotency key. State can be sent as a bounded payload or uploaded through a short-lived, scoped upload URL when the snapshot is large.

A successful request returns a snapshot identifier, creation time, content digest, parent snapshot when the backup is incremental, and a status URL. The agent records that receipt in its own journal. Repeating the same idempotent request returns the original result rather than creating an uncontrolled series of snapshots.

Restore is deliberately more restrictive. An MCP restore tool or POST /v1/restores request names the snapshot and target environment, but the service only releases state after checking caller authority, restore policy, compatibility, and any required operator approval.

Copying memory while an agent continues to dispatch tools can produce a snapshot that never represented a valid point in time. A practical pattern is a freeze-and-serialise loop keyed to sub-task boundaries. When the agent completes a discrete unit of work, the runtime enters a brief checkpoint barrier. This can occur, for example, after the agent reads a document but before it writes the summary. The runtime stops accepting new work, records the status of in-flight tool calls, flushes durable memory and the event journal, serialises the four state layers into a portable format, and assigns a monotonic checkpoint sequence. Work resumes once the state has been accepted or the checkpoint attempt has failed cleanly.

Checkpoint frequency is a cost decision as much as a safety one. Context windows can run to millions of tokens, and shipping a full copy after every step is computationally expensive and heavy on bandwidth and storage. Incremental snapshots record only the change from a parent. Checkpoints placed at consequential boundaries, rather than only on a timer, keep the recovery point current. Compacting older snapshot chains avoids storing a full copy each time.

Not every integration can pause. For those systems, the manifest records open operations and their reconciliation strategy. A restore can then query an external idempotency key, observe the completed result, retry a safe operation, or escalate an ambiguous side effect instead of blindly replaying it.

Snapshots are encrypted in transit and at rest, isolated by tenant, and protected by narrowly scoped workload identities. Envelope encryption allows each snapshot to use a data key protected by a service-managed or customer-managed key. Deleting or disabling the wrapping key can provide a separate control over whether retained state remains recoverable.

The manifest is canonicalised and signed. On restore, the service verifies the signature and the digest of every component before making state available. Audit records capture the requesting identity, policy decision, snapshot version, target environment, and approval without copying sensitive state into the audit stream.

Retention is a policy rather than a storage default. Operators define how many snapshots to keep, when to expire them, whether copies may cross regions, and which classifications of data may be backed up at all.

Recovery reconstructs state in a controlled runtime. The restore controller creates a clean runtime with a compatible agent package, validates the snapshot schema, verifies its contents, and applies any tested schema migration. It then reconstructs the saved context from the supported working-memory fields, reconnects episodic memory, and sets the execution cursor to the recorded checkpoint in the task tree. Tool grants and credentials are resolved afresh for the target environment; a backup never grants access merely because the original agent had it.

The restored agent begins in a suspended state. It can inspect its recovery metadata, reconcile incomplete operations, and produce a proposed resume point before an operator or policy permits tool use. This is especially important when the restore target has different network boundaries, data residency, model availability, or approval requirements.

A completed restore produces its own signed receipt linking the source snapshot, target runtime, migration version, reconciliation result, and activation decision. That record makes recovery observable and gives later investigators a clear continuity trail.

Restoring an agent does not restore the external systems that it used, so recovery requires reconciliation. Suppose the agent failed while issuing an invoice through a payment API. After restoration it cannot simply replay the call, because the original request may or may not have completed. The restored agent queries the provider using the idempotency key or reference recorded before the failure. It then retries, skips, or escalates the operation according to the observed state. Every side-effecting integration needs an idempotency, observation, or human-reconciliation path. Without one, recovery can recreate internal state while duplicating external consequences.

Long restore intervals create a related risk: stale memory. An agent restored from a two-week-old snapshot may not know that a project has finished, a colleague has left, or a policy has changed. A restore procedure should therefore include state validation, during which the agent compares its recorded state against current sources of truth before it acts.

A split-brain condition occurs if a network partition makes the original agent unreachable rather than stopping it. Restoring from backup can then produce two runtimes that pursue the same task independently. The service prevents concurrent writes with a liveness and fencing protocol. Each restore increments a generation number, and external systems and the agent's journal reject writes from superseded generations. Activation requires confirmation that the prior runtime has stopped or been isolated.

A snapshot supports operational continuity. The service can prove the provenance and integrity of recorded state, but it cannot reconstruct omitted context or guarantee identical behaviour after models, tools, prompts, or external data change.

Backup must not become a new single point of failure. Agents keep a local durable journal until a snapshot receipt is verified, retry transient failures with bounded backoff, and continue or stop according to workload policy. The service can replicate encrypted snapshots across approved failure domains while keeping region and residency constraints explicit.

Recovery objectives are measured per workload. Snapshot frequency determines the recovery point, while provisioning, verification, migration, and reconciliation determine recovery time. A high-value workflow may need incremental snapshots after every consequential action; a research agent may accept a longer interval and lower storage cost.

A successful backup request proves only that bytes were stored. The operational test restores the snapshot into an isolated environment, then checks memory, task state, policy, tool boundaries, and reconciliation behaviour. Scheduled restoration tests identify missing dependencies and schema drift before an incident.

The evaluation suite should cover interrupted uploads, corrupt components, revoked keys, expired snapshots, incompatible runtime versions, cross-tenant access attempts, and repeated restore requests. It should also test deliberately stale snapshots, simulated partitions that leave the original agent alive, and failures during external-operation reconciliation. The system is ready when those cases fail safely and leave useful evidence for an operator.