Recoverable agents: state, retries, and idempotency

A recoverable agent begins with durable task state. Conversation history describes what was said; task state describes where execution stands. They cannot replace one another.

After a restart, the system should know what completed, what remains uncertain, and whether another attempt is safe.

Make states explicit

A minimal set can include queued, running, awaiting confirmation, succeeded, failed, and cancelled. Transitions need conditions so an old request cannot move a completed task back into execution.

Store the task identifier, version, current step, necessary result references, and error category. Reconstruct model context from this state instead of treating an entire chat transcript as the only database.

Long waits also need timeouts or leases so an executor that disappeared cannot own a task forever.

Use a version to constrain competing execution

Update with an expected state and version, then inspect the affected-row count. This SQLite example demonstrates transactions and conditional updates in memory. An in-memory database does not survive a process restart; a real service needs persistent storage.

import sqlite3

db = sqlite3.connect(":memory:")
db.execute("CREATE TABLE tasks (id TEXT PRIMARY KEY, state TEXT, version INTEGER)")
db.execute("INSERT INTO tasks VALUES (?, ?, ?)", ("demo-task", "queued", 0))
db.commit()

with db:
    first = db.execute(
        "UPDATE tasks SET state='running', version=version+1 "
        "WHERE id=? AND state='queued' AND version=?",
        ("demo-task", 0),
    )
assert first.rowcount == 1

with db:
    duplicate = db.execute(
        "UPDATE tasks SET state='running', version=version+1 "
        "WHERE id=? AND state='queued' AND version=?",
        ("demo-task", 0),
    )
assert duplicate.rowcount == 0
assert db.execute("SELECT state, version FROM tasks").fetchone() == ("running", 1)
print("Only one claimant advanced this state")
db.close()

This prevents another claimant from advancing the same state. It does not prove that an external operation executes once. A local database transaction does not automatically cover a network request to another service.

Handle the gap between execution and persistence

An external write may finish before a process crashes while saving its result. Recovery can still see “running,” and blindly repeating the write can duplicate the action.

Use a stable idempotency key for a logical operation and verify that the external service supports it. Store request identifiers and result references, and query operation state when possible. Sending a request is not evidence of successful execution.

When the service cannot deduplicate or report status, uncertain results may need confirmation or reconciliation. A local state table alone does not provide end-to-end exactly-once execution.

Bound retries

Transient network errors may permit another attempt within time and attempt budgets. Permission failures, invalid input, and unconfirmed writes usually require a different branch. Reuse the same logical operation identifier instead of creating a new action on each attempt.

Retain the error category, attempt count, next execution time, and cancellation state. A recovery worker can use these explicit fields rather than guessing events from conversation text.

Try it next

Draw the state transitions for one task and annotate each edge with its conditions. Simulate interruption after claiming it, then check that a stale version cannot incorrectly reclaim the task.

Before adding an external write, verify its idempotency and status-query support and define an uncertain-result state. These boundaries provide the foundation for recoverable long-running execution.