Durable execution

An event history replayed after a crash: recorded steps are not run again, a step that reads the clock breaks the replay, and an idempotency key stops a double charge.

Loading the animation…

Concept

A checkpoint after every super-step is one way to survive a crash. Durable execution engines such as Temporal take the idea to its end: a workflow is ordinary code whose every call to the outside world (an activity) and its result are appended to an event history. When the worker running it dies, another worker replays the code from the top against the history: each call that is already recorded returns its recorded result at once, without running, and execution goes live where the history ends.

The animation runs an order workflow: reserve stock, charge the card £40, choose a shipping option, ship, send a receipt. Inject a fault and watch the history and the outside world:

Crash after the charge. The charge's result is in the history. The new worker replays, the charge comes back from the history, and the card is charged once (1).
Crash in the charge. The payment service took the money, but the worker died before the result was recorded. On replay the history shows the charge as scheduled but not completed, so it must run again: execution is at least once. Without a key the card is charged 2 times (£80). With an idempotency key derived from the workflow and the step (order-1042/1), the service recognises the second request and returns the first result: one charge. Exactly-once effects = at-least-once execution + idempotent effects.
Lost reply. The same without a crash: the charge succeeds but its reply is lost; after a 2,000 ms timeout the worker retries. 2 charges without a key, 1 with one.
Crash in shipping, with the clock read in the code. The code chooses express shipping because the clock says it is early. The replay runs later, the same line now chooses standard, and the command no longer matches the history: a non-determinism error at the first command that differs, and the workflow is stuck until the code is fixed. Record the choice in the history (a marker: the role Temporal's SideEffect call plays) and the replay reads it back: the run completes. Temporal's documentation puts the rule plainly: workflow code "must be deterministic to support replay".

LangGraph's checkpoints play the history's part, and the same rules apply to its nodes: a node that is re-run after a crash or an interrupt repeats its side effects, so make them idempotent, and keep anything that varies between runs (the clock, random numbers, a model's sampled answer) in the state rather than recomputed. Its durability setting chooses when the writes are persisted: "sync" before the next step starts, "async" while the next step runs, "exit" only when the graph exits (the Durability type in langgraph 1.2.14's types.py).

Maths

Let each attempt's reply be lost independently with probability $q$ , and allow $A$ attempts. Without a key every attempt charges, and attempt $i+1$ happens only if the first $i$ replies were lost, so the expected number of charges is the geometric sum in the equation; the card is charged at least twice exactly when the first reply is lost, probability $q$ (for $A \ge 2$ ), and the order fails with every reply lost, probability $q^A$ . With a key the service charges once whatever happens to the replies. At $q = 0.2$ and $A = 3$ the sum is $1 + 0.2 + 0.04 = 1.24$ ; the seeded sweep in the widget gives the simulated value beside each closed form, and the engine's tests require them to agree within four standard errors.

Replay is deterministic only if the code is: replaying history $H = (e_0, e_1, \ldots)$ must issue the same command sequence the first run did, $\mathrm{cmd}_i(\text{code}, H_{<i}) = e_i$ for every recorded command. Reading the clock makes $\mathrm{cmd}_i$ depend on when the replay runs; recording the value as a marker makes it depend only on $H$ .

Code

The service's check, and the replay's check, cut from the vendored engine (durable.ts):

if (key !== null) for (const c of ledger) if (c.key === key) return "dedup";

if (ev.type !== "activity_scheduled" || ev.activity !== name) return nondet(ev, `schedule ${name}`, line, cursor);