Design an agent workflow that can resume safely after a crash

The crash you never tested for

An AI coding agent is halfway through a multi-step task: it has read a codebase, generated a migration file, applied it to a staging database, and is about to open a pull request. The process dies. Maybe the container was evicted, an API call timed out, or the host ran out of memory.

When the operator restarts the agent, three things can happen. The good outcome: the agent picks up where it left off. The annoying outcome: it starts from scratch, re-reads the codebase, and re-generates the migration -- wasting time but causing no harm. The dangerous outcome: it applies the migration a second time, creating duplicate columns or corrupting data, because nothing recorded that the migration already succeeded.

Most agent frameworks focus on the conversation loop -- prompt in, response out, append to history. They do not distinguish between three fundamentally different kinds of state that a crash can disrupt:

• Conversation history: the sequence of messages exchanged with the model. Losing this is wasteful but rarely dangerous.
• Workflow state: which stage the agent has reached, what decisions it has made, and what it plans to do next. Losing this causes confusion and repeated work.
• Completed side effects: external writes that have already happened -- database changes, API calls, file mutations, deployed artifacts. Losing track of these causes duplication and corruption.

This guide is for engineers who operate or build long-running agent workflows and need those workflows to resume correctly. It provides a concrete state model, a failure-injection matrix, and a resume checklist you can adapt to your own system.

Quick-start path

1. List every external side effect your agent can perform. Classify each as idempotent or non-idempotent.
2. Add a durable checkpoint after each stage transition. Store it outside the agent process.
3. Assign an attempt identity (a unique ID) to each task run. Propagate it into every external call.
4. For non-idempotent effects, record completion in the checkpoint store before acknowledging success to the next stage.
5. On restart, read the last checkpoint, skip completed stages, and verify pending effects before retrying.
6. Run the failure-injection matrix (below) against your workflow at least once before production use.

Three kinds of state and why each needs different durability

Conversation history

This is the token sequence the model has seen. It matters for coherence but not for correctness. If you lose conversation history, the agent may generate slightly different output on retry, but no external system is corrupted. Store it if you want continuity; do not treat it as your crash-recovery mechanism.

Workflow state

This is the agent's position in a directed graph of stages. A hypothetical coding agent might have stages like: CLAIM_TASK -> READ_CODEBASE -> GENERATE_PATCH -> RUN_TESTS -> OPEN_PR -> DONE. Workflow state also includes decisions made at branch points -- for example, "chose migration strategy A over B."

Workflow state must be durable. Store it in a database, a file on durable storage, or an external orchestration service. The checkpoint must be written atomically: either the full stage transition is recorded, or none of it is.

Completed side effects

These are mutations to the outside world: a row inserted, an API called, a file written, a PR opened. The critical rule is: you must record that a side effect completed before you advance the workflow stage. If the process dies between performing the effect and recording it, the recovery path must be able to detect whether the effect already happened.

This is why attempt identities matter. If every migration your agent applies carries a unique ID, the recovery logic can check whether that migration ID already exists before applying it again.

Stage checkpoints, attempt identities, claim leases, and stale-worker fencing

Stage checkpoints

A checkpoint is a durable record of: the current stage, the attempt identity, a timestamp, and any outputs from completed stages that downstream stages need. Write a checkpoint on every stage transition, not just at the end.

Think of checkpoints as the "save game" for your workflow. Without them, a restart means replaying from the beginning -- and replaying side effects is where things break.

Attempt identities

Every time a workflow begins or resumes, generate a unique attempt ID (a UUID or similar). Pass this ID into every external call. This gives you two things:

• Idempotency keys: external APIs that support idempotency keys can deduplicate calls automatically.
• Audit trail: when something goes wrong, you can trace which attempt caused which effect.

Claim leases

If multiple workers might pick up the same task (common in queue-based architectures), you need a claim lease: a time-bounded lock that says "worker W owns task T until time X." If the worker crashes, the lease expires and another worker can claim the task. The new worker reads the last checkpoint, not the conversation history, to decide where to resume.

Lease duration is a tradeoff. Too short: a slow but healthy worker loses its lease and a second worker starts, causing duplicate work. Too long: a crashed worker blocks progress. A reasonable starting point for most agent workflows is 2-5x the expected duration of the longest single stage.

Stale-worker fencing

A lease alone is not enough. Consider: Worker A's lease expires. Worker B claims the task and starts from the last checkpoint. Worker A, which was merely slow (not dead), finishes its current stage and writes a result. Now you have two workers writing to the same workflow.

Fencing prevents this. The simplest approach: include a lease generation number (or epoch) in every write. The checkpoint store rejects writes from older generations. Worker A's write fails because its generation is stale. Worker B proceeds safely.

Three crash scenarios walked through

The following hypothetical uses a coding agent that performs four stages: (1) read repository, (2) generate a patch, (3) apply the patch to a branch, (4) open a pull request.

Crash before a write (between stages 1 and 2)

The agent has read the repository and is generating a patch. The process dies before any external mutation occurs. On restart, the new worker reads the checkpoint, sees that stage 1 completed, and re-enters stage 2. No external state was changed, so there is nothing to deduplicate. The only cost is re-running the LLM call, which may produce a slightly different patch. This is safe.

Why this is the easy case: no side effects means no duplication risk. The only decision is whether you care about determinism (getting the exact same patch). If you do, you need to cache the model output as part of the stage-1 checkpoint.

Crash after a write, before the next checkpoint (during stage 3)

The agent has pushed a commit to a branch. The process dies before recording that stage 3 completed. On restart, the new worker reads the checkpoint, sees stage 2 completed, and enters stage 3 again. It tries to push the same commit.

If the agent used an attempt-specific branch name (e.g., agent/task-42/attempt-7a3b), the push will either succeed idempotently (same content, same ref) or fail visibly (ref already exists with that content). The recovery logic checks: does the expected branch with the expected commit already exist? If yes, mark stage 3 complete and advance. If no, retry the push.

If the agent used a shared branch name with no attempt identity, the second push might create a duplicate commit, a merge conflict, or silently overwrite. This is the dangerous case that attempt identities prevent.

Why this is the hard case: the side effect happened, but the record of it did not. Your recovery code must be able to detect the completed effect from external evidence.

Crash after a write, before acknowledgment (during stage 4)

The agent calls the API to open a pull request. The API responds with 201 Created. The process dies before the agent processes the response and writes the checkpoint. On restart, the new worker enters stage 4 and tries to open the PR again.

If the PR creation call included an idempotency key (or the recovery logic searches for an existing PR from the same attempt), the duplicate is caught. If not, you get two identical pull requests.

Key insight: "before acknowledgment" crashes are functionally identical to "after a write" crashes from the workflow's perspective. The effect happened; the workflow does not know it. The mitigation is the same: external evidence checks or idempotency keys.

Why retrying a prompt differs from retrying an external action

Retrying a prompt -- sending the same messages to the model again -- is almost always safe. The model has no persistent state that changes between calls. The output may differ (non-determinism), but no external system is corrupted by a redundant prompt.

Retrying an external action -- inserting a database row, calling a third-party API, deploying an artifact -- can be dangerous. The external system may have state that changes on each call. Inserting a row twice creates duplicates. Deploying twice may trigger redundant notifications or billing.

This distinction matters because many agent frameworks treat the entire loop uniformly: prompt, parse, act, repeat. When the loop restarts after a crash, the framework retries everything from the last message boundary, including actions. The fix is to separate the "think" steps (safe to retry) from the "act" steps (must be idempotent or deduplicated).

A practical rule: wrap every external action in an idempotency guard before exposing it as a tool the agent can call. The guard checks whether the action already completed for this attempt ID. If yes, it returns the cached result. If no, it performs the action and records the result durably.

Failure-injection matrix and resume checklist

Failure-injection matrix (copyable)

Use this matrix to systematically test crash recovery. For each cell, simulate the failure, restart the workflow, and verify the expected behavior. Fill in your own stages and effects.

Instructions: Copy this table. Replace the example stages and effects with your workflow's actual stages. For each cell, write "TESTED" and the date after you have injected that failure and confirmed correct behavior. Blank cells indicate untested recovery paths.

FAILURE-INJECTION MATRIX
=========================
Workflow: [your workflow name]
Date: [date of last full pass]

Stage               | Failure Point             | Side Effect at Risk     | Expected Recovery Behavior                    | Tested?
--------------------|---------------------------|-------------------------|-----------------------------------------------|--------
Read repository     | Process kill mid-read     | None                    | Restart from stage beginning                  | [    ]
Generate patch      | OOM during inference      | None (no external write)| Re-run generation from checkpoint              | [    ]
Apply patch (push)  | Kill after git push,      | Branch may exist with   | Check if branch+commit exist; skip if match;  | [    ]
                    | before checkpoint write    | expected commit         | re-push if not                                |
Open pull request   | Kill after API 201,       | PR may already exist    | Search for PR by attempt ID or branch;        | [    ]
                    | before checkpoint write    |                         | skip if found; create if not                  |
Open pull request   | API timeout (unknown      | PR may or may not exist | Same as above; treat timeout as "unknown"     | [    ]
                    | whether 201 was received) |                         | and check before retrying                     |
Any stage           | Lease expiry while        | Concurrent worker may   | Fencing token rejects stale writes;           | [    ]
                    | worker is slow            | start                   | only current-generation worker proceeds       |
Any stage           | Checkpoint store           | Workflow cannot record  | Worker retries checkpoint write with backoff;  | [    ]
                    | temporarily unavailable   | progress                | does NOT proceed to next stage without record |

Resume checklist (copyable)

Run this checklist every time a workflow is resumed after an unexpected stop.

RESUME CHECKLIST
================
Task ID: _______________
Previous Attempt ID: _______________
New Attempt ID: _______________

[ ] 1. Read the last durable checkpoint. Record the completed stage: _______________
[ ] 2. Verify the checkpoint store is writable (test write + delete).
[ ] 3. For the last completed stage, confirm the side effect matches expectations:
       - What effect should exist? _______________
       - Does it exist in the external system? [ YES / NO / CANNOT DETERMINE ]
       - If CANNOT DETERMINE, escalate to a human before proceeding.
[ ] 4. For the next stage, confirm no partial effect exists:
       - Expected state of external system before this stage: _______________
       - Actual state: _______________
       - Match? [ YES / NO ]
       - If NO, assess whether the partial effect is safe to retry or requires cleanup.
[ ] 5. Acquire a new claim lease with the new attempt ID.
[ ] 6. Confirm the fencing token/generation is set for this attempt.
[ ] 7. Begin execution from the next incomplete stage.
[ ] 8. After completion, verify all side effects against expected outcomes.

Hypothetical worked scenario: a coding agent resumes after a container eviction

Setup: A hypothetical team runs an AI coding agent that picks tasks from a queue, generates code changes, and opens PRs. The agent runs in a container orchestrator. Task #312 involves adding input validation to an API endpoint.

What happens:

1. Worker A claims task #312 with attempt ID att-9f2c. It reads the repo (stage 1), writes a checkpoint {task: 312, attempt: att-9f2c, stage: READ_COMPLETE, timestamp: 14:01}.
2. Worker A generates a patch (stage 2), writes a checkpoint {stage: PATCH_GENERATED, patch_hash: abc123}.
3. Worker A pushes a commit to branch agent/312/att-9f2c (stage 3). The push succeeds. Before the checkpoint is written, the container is evicted due to a resource limit.
4. The claim lease (set to 10 minutes) expires at 14:15. Worker B picks up task #312 from the queue with new attempt ID att-d471.
5. Worker B reads the last checkpoint: PATCH_GENERATED. It enters stage 3 (apply patch).
6. Worker B checks: does branch agent/312/att-9f2c exist with commit hash abc123? Yes, it does. Worker B writes a checkpoint {stage: PATCH_APPLIED, branch: agent/312/att-9f2c} and advances.
7. Worker B opens a PR, records the PR URL in the checkpoint, and completes the workflow.
8. Suppose Worker A's container is rescheduled and it tries to resume. It attempts to write a checkpoint, but its fencing generation is stale. The write is rejected. Worker A logs the rejection and exits cleanly.

Outcome: No duplicate branch, no duplicate PR, no data corruption. The total cost was re-running the branch existence check, which took milliseconds.

What would have gone wrong without these safeguards: Worker B would have re-pushed the commit (possibly creating a duplicate or conflicting branch), then opened a second PR. Worker A, on resumption, might have opened a third. The team would have found three PRs for the same task.

Comparing realistic options for crash recovery

Teams using FeatureFactory to examine engineering delivery across AI-assisted and human-authored work may find that agent crash-recovery behavior surfaces as unexplained rework or duplicate PRs. Understanding which workflows lack durable state helps explain those signals.

Failure symptoms, corrective actions, and escalation points

Explicit limitations

• This guide does not cover distributed transactions. If your agent performs writes across multiple external systems that must be atomically consistent, you need a saga pattern or two-phase commit, which is beyond the scope of this guide.
• Idempotency is not free. Wrapping every action in an idempotency guard requires designing for it upfront. Retrofitting idempotency onto an existing agent tool set is labor-intensive.
• LLM non-determinism means retried prompts may diverge. Even with the same input, the model may produce a different patch on retry. If your workflow requires deterministic outputs, you must cache the model's output in the checkpoint, not just the stage marker.
• Fencing depends on the checkpoint store supporting conditional writes. Not all storage systems do. If yours does not, you may need a separate lock service.
• The failure-injection matrix is a minimum, not a guarantee. Real production failures combine in ways a matrix cannot enumerate. The matrix catches the common single-fault cases.
• All numbers, durations, and scenarios in this guide are hypothetical. They illustrate mechanics, not measured benchmarks.

Adapting to different team contexts

Solo developer running a local agent: You may not need claim leases or fencing. You do still need checkpoints and idempotency guards if the agent performs any external writes. A JSON file on disk is a sufficient checkpoint store.

Small team with a shared task queue: Add claim leases. Use a database row with a version column for fencing. The failure-injection matrix becomes a team exercise -- each developer injects one failure type and documents the result.

Platform team providing agent infrastructure to many teams: Build the checkpoint store and idempotency guard as shared primitives. Individual workflow authors should not have to implement their own. Publish the resume checklist as a required part of workflow certification. When teams use product planning tools to organize agent-related work, include "crash-recovery tested" as a readiness signal.

Ongoing review routine

Run this review monthly, or after any incident involving agent resumption:

1. Check the failure-injection matrix. Are all cells tested for the current version of the workflow? Any new stages added since the last pass?
2. Review checkpoint store health metrics. How often do checkpoint writes fail or take longer than 1 second? Degraded checkpoint writes are a leading indicator of future resume failures.
3. Audit duplicate side effects. Search for duplicate PRs, branches, or database writes created by agent workflows. Any duplicates indicate a gap in idempotency guards.
4. Review lease duration against actual stage durations. If stages have gotten slower (larger repos, more complex tasks), leases may need adjustment.
5. Confirm fencing is active. Periodically verify that stale-generation writes are actually rejected, not silently accepted due to a misconfiguration.

Teams that examine delivery patterns across agent-assisted workflows can use duplicate-detection trends as an input to this review.

Next-action checklist

• Inventory every external side effect your agent can perform. Write each one down.
• Classify each effect as idempotent or non-idempotent. For non-idempotent effects, design an idempotency guard using attempt IDs.
• Choose a checkpoint store. Verify it supports atomic conditional writes (for fencing).
• Implement stage checkpoints. Test that a kill -9 between any two stages results in correct resumption.
• If using multiple workers, implement claim leases and fencing tokens.
• Fill in the failure-injection matrix with your actual workflow stages and run each test.
• Run the resume checklist on your next real crash (or simulated one) and record the results.
• Schedule the first monthly review. Put it on the calendar now; do not wait for an incident.
• Share this guide and your completed matrix with anyone who operates or modifies the agent workflow. Recovery logic only works if the people changing the workflow know it exists.