← Knowledge

Recoverable Agent Execution

How agent work resumes after retries, interruption, duplicate delivery, and partial failure.

Recoverable agent execution records enough state for interrupted work to resume without guessing what happened. The design applies distributed-systems recovery methods to agent work that can span retries, duplicate messages, process restarts, or several scheduled runs.

Persist progress before effects

A recoverable workflow records its state wherever work can fail or an external effect can occur. Before sending a message, charging an account, changing a repository, or starting a remote job, the workflow should store enough intent to recognize the same operation later. After the effect completes, it should store a receipt from the system that performed it.

The difficult case is an effect that succeeds before its receipt is stored. Recovery code must query the external system, deduplicate the operation, or reconcile the missing receipt instead of repeating the action blindly.

Give retries an identity

An operation is idempotent when repeating it produces the same intended state instead of duplicating the effect. Retry logic alone does not provide that property. Stable operation identifiers, compare-and-swap conditions, content digests, and provider idempotency keys are common mechanisms.

A timeout is also a state transition. It does not prove that nothing happened. Once a timeout becomes authoritative, a late success must be reconciled explicitly. Otherwise the workflow can report failure while the outside system records success.

Store receipts for observed results

Receipts distinguish planned work from completed work. Useful receipts include message identifiers, transaction versions, commit hashes, deployment releases, test results, and timestamps from the system that performed the effect.

Agent Authority and Effects separates the permission to attempt an action from the evidence that the action occurred. Agent Trajectory Observability connects that receipt to the model run and tool call that produced it.

Record the execution context

The model, code, tools, memory, or policy may change before a later resumption. Record the workflow version and the relevant context assumptions used at each step. Spec-driven development preserves intent and verification criteria; recoverable execution preserves the progress state that connects those artifacts over time.

Persistent runtimes such as Letta Code can schedule and resume work. Recovery still depends on explicit state transitions and external receipts rather than the mere existence of a long-lived agent.

Sources

  1. Temporal workflow execution
  2. DBOS documentation

Connections

Related

Linked here

Suggest a correction ↗