/data

Conformance and results

Everything here is computed by the vendored engine when the site is built, from the engine at commit 46b445c; the engine's own recorded runs are orchestration_results.md (chapters 1–4) and orchestration2_results.md (chapters 5–9). Every token size, accuracy, rate limit, failure rate and timing in chapters 5–9 is illustrative.

LangGraph conformance

Recorded with langgraph 1.2.14 (langgraph-checkpoint 4.2.0, an in-memory checkpointer) by record_langgraph.py from graphs built by langgraph_graphs.py; the recording is langgraph_recordings.json. After every operation the full checkpoint history (step, source, parent, state, next tasks, each task's result, error and interrupts), the outcome, the node runs and the side effects are compared with the engine's; the engine's CI re-records them live on every change. 39 sessions, 71 operations, 232 checkpoints in the final histories.

session	graph	operations	outcomes	checkpoints
chain	chain	invoke	done	5
fanout	fanout	invoke	done	5
react_loop	react_loop	invoke	done	7
uneven_join	uneven_join	invoke	done	6
no_join	no_join	invoke	done	5
router_view	router_view	invoke	done	3
merge_overwrite	merge_overwrite	invoke	error (InvalidUpdateError)	3
merge_add	merge_add	invoke	done	5
merge_max	merge_max	invoke	done	5
fanout_list	fanout_list	invoke	done	4
start_router	start_router	invoke	done	3
approval	approval	invoke; resume	interrupted; done	5
approval_no	approval	invoke; resume	interrupted; done	5
gated	gated	invoke; continue	paused; done	5
gated_edit	gated	invoke; update 2 as planner; continue	paused; updated; done	6
gated_after	gated_after	invoke; continue	paused; done	4
crash	crash	invoke (crash enrich); continue	error (NodeFailure); done	5
crash_twice	crash	invoke (crash enrich); continue (crash enrich); continue	error (NodeFailure); error (NodeFailure); done	5
counter_ok	counter	invoke (limit 5)	done	6
counter_edge	counter	invoke (limit 5)	recursion_limit	7
counter_over	counter	invoke (limit 5); continue (limit 3); continue (limit 25)	recursion_limit; done; done	10
replay	branchy	invoke; replay 2	done; done	8
fork	branchy	invoke; update 2 as plan; continue	done; updated; done	8
fork_other_node	branchy	invoke; update 2 as search; continue	done; updated; done	7
replay_update	branchy	invoke; update 2 as plan; replay 5	done; updated; done	8
loop_join	loop_join	invoke	done	8
start_router_else	start_router	invoke	done	3
loop_join_crash	loop_join	invoke; replay 3; update 5 as check; continue	done; done; updated; done	14
after_update	gated_after	invoke; update 2 as a; continue	paused; updated; done	5
replay_interrupt	gated	invoke; continue; replay 2	paused; done; done	8
map_reduce	map_reduce	invoke	done	5
map_reduce_crash	map_reduce	invoke (crash summarise); continue	error (NodeFailure); done	5
map_reduce_replay	map_reduce	invoke; replay 2	done; done	8
map_reduce_update	map_reduce	invoke; update 2 as split; continue	done; updated; done	8
map_reduce_none	map_reduce	invoke; update 2 as split; continue	done; updated; done	6
map_review	map_review	invoke; resume	interrupted; done	5
send_mixed	send_mixed	invoke	done	5
map_loop	map_loop	invoke	done	8
map_empty	map_loop	invoke	done	4

Super-steps of chapter 1's graphs

Illustrative node durations and a 15 ms checkpoint write; the super-steps are LangGraph's (the bulk-synchronous model of Pregel, SIGMOD 2010).

graph	super-steps	graph time	one node at a time	speed-up
chain	4	2.86 s	2.86 s	1.00×
fanout	4	3.26 s	4.47 s	1.37×
react_loop	6	3.59 s	3.59 s	1.00×
loop_join	7	2.9 s	3.94 s	1.35×

Durable execution runs (chapter 4)

Every combination the widget offers: fault, idempotency key, and how the shipping choice is made. The history's shape follows durable-execution engines such as Temporal, simplified; every time is illustrative.

fault \| key \| choice	outcome	charges	£ charged	events	finished at
none \| nokey \| clock	completed	1	40	10	1.8 s
none \| nokey \| marker	completed	1	40	11	1.8 s
none \| key \| clock	completed	1	40	10	1.8 s
none \| key \| marker	completed	1	40	11	1.8 s
crash_after_charge \| nokey \| clock	completed	1	40	10	6.81 s
crash_after_charge \| nokey \| marker	completed	1	40	11	6.81 s
crash_after_charge \| key \| clock	completed	1	40	10	6.81 s
crash_after_charge \| key \| marker	completed	1	40	11	6.81 s
crash_in_charge \| nokey \| clock	completed	2	80	10	7.61 s
crash_in_charge \| nokey \| marker	completed	2	80	11	7.61 s
crash_in_charge \| key \| clock	completed	1	40	10	7.61 s
crash_in_charge \| key \| marker	completed	1	40	11	7.61 s
crash_in_ship \| nokey \| clock	nondeterminism	1	40	6	6.61 s
crash_in_ship \| nokey \| marker	completed	1	40	11	7.31 s
crash_in_ship \| key \| clock	nondeterminism	1	40	6	6.61 s
crash_in_ship \| key \| marker	completed	1	40	11	7.31 s
lost_reply \| nokey \| clock	completed	2	80	11	4.6 s
lost_reply \| nokey \| marker	completed	2	80	12	4.6 s
lost_reply \| key \| clock	completed	1	40	11	4.6 s
lost_reply \| key \| marker	completed	1	40	12	4.6 s

Lost replies and retries: simulation against closed form

Each charge attempt's reply lost with probability q, at most A attempts, every attempt reaching the service. Seeded (mulberry32), the same in Python and in the browser.

q	A	mean charges, no key	(1 − q^A)/(1 − q)	P(two or more)	q	P(all lost)	q^A	with a key
0.05	1	1.0000	1.0000	0.0000	0.0000	0.0488	0.0500	1.0000
0.05	2	1.0491	1.0500	0.0491	0.0500	0.0023	0.0025	1.0000
0.05	3	1.0514	1.0525	0.0492	0.0500	0.0001	0.0001	1.0000
0.05	5	1.0515	1.0526	0.0491	0.0500	0.0000	0.0000	1.0000
0.1	1	1.0000	1.0000	0.0000	0.0000	0.0969	0.1000	1.0000
0.1	2	1.0970	1.1000	0.0969	0.1000	0.0097	0.0100	1.0000
0.1	3	1.1064	1.1100	0.0969	0.1000	0.0011	0.0010	1.0000
0.1	5	1.1076	1.1111	0.0968	0.1000	0.0000	0.0000	1.0000
0.2	1	1.0000	1.0000	0.0000	0.0000	0.1983	0.2000	1.0000
0.2	2	1.2001	1.2000	0.2001	0.2000	0.0399	0.0400	1.0000
0.2	3	1.2411	1.2400	0.2006	0.2000	0.0074	0.0080	1.0000
0.2	5	1.2494	1.2496	0.1999	0.2000	0.0006	0.0003	1.0000
0.3	1	1.0000	1.0000	0.0000	0.0000	0.2998	0.3000	1.0000
0.3	2	1.2995	1.3000	0.2995	0.3000	0.0903	0.0900	1.0000
0.3	3	1.3912	1.3900	0.3013	0.3000	0.0259	0.0270	1.0000
0.3	5	1.4247	1.4251	0.3005	0.3000	0.0027	0.0024	1.0000
0.5	1	1.0000	1.0000	0.0000	0.0000	0.4995	0.5000	1.0000
0.5	2	1.4970	1.5000	0.4970	0.5000	0.2524	0.2500	1.0000
0.5	3	1.7528	1.7500	0.4992	0.5000	0.1241	0.1250	1.0000
0.5	5	1.9387	1.9375	0.4978	0.5000	0.0321	0.0313	1.0000

20 settings of 20,000 orders each, seed 24.

Multi-agent patterns (chapter 5)

Uncontended (no queueing), Claude Sonnet 4.6 list prices, the hosted latency profile; accuracy falls 0.01 per 1,000 prompt tokens past 4,000 (illustrative). Sub-questions × tool-result tokens.

task	pattern	model calls	input tokens	cached	cost	latency	P(success)
4 × 700	single	5	12,700	7,620	$0.0309	14.8 s	90.3%
4 × 700	supervisor	13	19,640	9,120	$0.0704	34.6 s	81.7%
4 × 700	hierarchical	17	23,590	10,350	$0.0902	44.5 s	77.7%
4 × 700	swarm	8	24,040	9,480	$0.081	29.6 s	84.0%
4 × 700	debate	11	37,020	16,500	$0.11	23.1 s	90.2%
4 × 700	map_reduce	10	14,480	4,200	$0.068	15.9 s	83.0%
4 × 1500	single	5	20,700	12,420	$0.0444	15.5 s	85.7%
4 × 1500	supervisor	13	22,840	9,120	$0.0824	35.3 s	81.7%
4 × 1500	hierarchical	17	26,790	10,350	$0.10	45.2 s	77.7%
4 × 1500	swarm	8	36,840	14,280	$0.11	31.2 s	76.4%
4 × 1500	debate	11	61,020	28,500	$0.16	24.2 s	87.8%
4 × 1500	map_reduce	10	17,680	4,200	$0.08	16 s	83.0%
8 × 700	single	9	36,540	28,420	$0.0522	23.2 s	76.8%
8 × 700	supervisor	25	43,220	24,220	$0.13	1 min 3 s	68.1%
8 × 700	hierarchical	29	43,570	21,850	$0.15	1 min 13 s	64.8%
8 × 700	swarm	16	82,320	37,100	$0.23	1 min 1 s	54.4%
8 × 700	debate	15	76,060	43,380	$0.17	32.1 s	78.6%
8 × 700	map_reduce	18	26,920	8,400	$0.12	19.1 s	70.6%
8 × 1500	single	9	65,340	50,820	$0.0829	24.5 s	58.3%
8 × 1500	supervisor	25	49,620	24,220	$0.15	1 min 4 s	68.1%
8 × 1500	hierarchical	29	49,970	21,850	$0.17	1 min 14 s	64.8%
8 × 1500	swarm	16	133,520	59,500	$0.34	1 min 6 s	32.6%
8 × 1500	debate	15	136,860	79,380	$0.28	34.5 s	61.9%
8 × 1500	map_reduce	18	33,320	8,400	$0.15	19.2 s	70.6%
16 × 700	single	17	120,700	106,500	$0.11	40.1 s	37.6%
16 × 700	supervisor	49	105,260	69,300	$0.25	1 min 59 s	46.4%
16 × 700	hierarchical	53	90,970	52,290	$0.26	2 min 9 s	45.1%
16 × 700	swarm	32	301,600	143,700	$0.73	2 min 13 s	7.4%
16 × 700	debate	23	190,620	133,620	$0.30	50.1 s	39.9%
16 × 700	map_reduce	34	51,800	16,800	$0.23	25.5 s	50.6%
16 × 1500	single	17	229,500	202,500	$0.18	42.6 s	11.2%
16 × 1500	supervisor	49	118,060	69,300	$0.30	2 min 2 s	46.4%
16 × 1500	hierarchical	53	103,770	52,290	$0.31	2 min 12 s	45.1%
16 × 1500	swarm	32	506,400	239,700	$1.16	2 min 35 s	0.7%
16 × 1500	debate	23	363,420	256,020	$0.52	55.1 s	12.3%
16 × 1500	map_reduce	34	64,600	16,800	$0.28	25.6 s	50.6%

Fan-out under a rate limit (chapter 6)

A map-reduce over 32 chunks of 2000 tokens, 50 requests and 8,000 output tokens a minute, simulated with no failures.

input tokens / min	width	run time	speed-up	Amdahl	bucket bound
40,000	1	2 min 21 s	1.00×	1.00×	1.49×
40,000	2	1 min 42 s	1.38×	1.88×	1.49×
40,000	4	1 min 42 s	1.38×	3.34×	1.49×
40,000	8	1 min 42 s	1.38×	5.49×	1.49×
40,000	16	1 min 42 s	1.38×	8.08×	1.49×
40,000	32	1 min 42 s	1.38×	10.58×	1.49×
80,000	1	2 min 21 s	1.00×	1.00×	8.20×
80,000	2	1 min 15 s	1.88×	1.88×	8.20×
80,000	4	42 s	3.34×	3.34×	8.20×
80,000	8	25.6 s	5.49×	5.49×	8.20×
80,000	16	25.5 s	5.50×	8.08×	8.20×
80,000	32	25.5 s	5.50×	10.58×	8.20×
200,000	1	2 min 21 s	1.00×	1.00×	36.74×
200,000	2	1 min 15 s	1.88×	1.88×	36.74×
200,000	4	42 s	3.34×	3.34×	36.74×
200,000	8	25.6 s	5.49×	5.49×	36.74×
200,000	16	17.4 s	8.08×	8.08×	36.74×
200,000	32	13.3 s	10.58×	10.58×	36.74×

Reliability: closed form against simulation (chapter 7)

Visible errors 0.05, silent errors 0.03 per attempt; 2,000 seeded runs a point; 95% Wilson intervals.

each step gets	steps	closed form	simulated	95% interval	attempts	closed form
one attempt	1	0.9200	0.9245	0.912–0.935	1.00	1.00
one attempt	2	0.8464	0.8580	0.842–0.873	1.96	1.95
one attempt	3	0.7787	0.7565	0.737–0.775	2.84	2.85
one attempt	5	0.6591	0.6610	0.640–0.681	4.51	4.52
one attempt	8	0.5132	0.4990	0.477–0.521	6.74	6.73
one attempt	10	0.4344	0.4380	0.416–0.460	7.91	8.03
one attempt	15	0.2863	0.2910	0.272–0.311	10.91	10.73
one attempt	20	0.1887	0.1715	0.156–0.189	12.65	12.83
one attempt	30	0.0820	0.0930	0.081–0.107	16.10	15.71
retries (3 attempts)	1	0.9683	0.9675	0.959–0.974	1.05	1.05
retries (3 attempts)	2	0.9376	0.9435	0.933–0.953	2.10	2.10
retries (3 attempts)	3	0.9079	0.9035	0.890–0.916	3.17	3.16
retries (3 attempts)	5	0.8512	0.8545	0.838–0.869	5.27	5.26
retries (3 attempts)	8	0.7728	0.7495	0.730–0.768	8.41	8.42
retries (3 attempts)	10	0.7246	0.7290	0.709–0.748	10.53	10.52
retries (3 attempts)	15	0.6168	0.5900	0.568–0.611	15.75	15.77
retries (3 attempts)	20	0.5250	0.5055	0.484–0.527	21.07	21.03
retries (3 attempts)	30	0.3805	0.4075	0.386–0.429	31.48	31.52
retries + verifier	1	0.9915	0.9920	0.987–0.995	1.13	1.13
retries + verifier	2	0.9830	0.9875	0.982–0.992	2.27	2.27
retries + verifier	3	0.9746	0.9715	0.963–0.978	3.40	3.40
retries + verifier	5	0.9581	0.9545	0.944–0.963	5.64	5.65
retries + verifier	8	0.9337	0.9320	0.920–0.942	9.04	9.02
retries + verifier	10	0.9179	0.9245	0.912–0.935	11.25	11.26
retries + verifier	15	0.8794	0.8805	0.866–0.894	16.84	16.81
retries + verifier	20	0.8425	0.8365	0.820–0.852	22.42	22.32
retries + verifier	30	0.7733	0.7825	0.764–0.800	33.18	33.19

Many workflows at once (chapter 8)

200 workflows per row, Poisson arrivals, seed 11; 50 requests, 40,000 input and 8,000 output tokens a minute, concurrency 16, transient failures 3% per attempt, up to 3 attempts.

pattern	capacity / min	arrivals / min	success	p50	p95	p99	$ per success	completed / min
single	7.87	1	92.5%	16.8 s	19.4 s	20.1 s	$0.0334	0.99
single	7.87	2	92.5%	17 s	19.2 s	20 s	$0.0334	1.98
single	7.87	3	92.0%	16.8 s	19.5 s	20.6 s	$0.0336	2.95
single	7.87	4	92.5%	16.6 s	18.8 s	20.2 s	$0.0334	3.95
single	7.87	6	92.5%	16.6 s	19.1 s	19.7 s	$0.0334	5.91
single	7.87	8	87.5%	1 min 33 s	2 min 39 s	2 min 58 s	$0.0353	6.97
supervisor	3.80	1	81.0%	39.6 s	42.8 s	44.5 s	$0.0869	0.87
supervisor	3.80	2	78.5%	39.9 s	43.4 s	45.2 s	$0.0896	1.68
supervisor	3.80	3	80.5%	40.5 s	1 min 6 s	1 min 15 s	$0.087	2.57
supervisor	3.80	4	85.0%	7 min 58 s	10 min 45 s	11 min 27 s	$0.0828	3.15
supervisor	3.80	6	83.5%	27 min 22 s	31 min 11 s	32 min 47 s	$0.0843	2.96
supervisor	3.80	8	83.5%	35 min 11 s	38 min 31 s	40 min 38 s	$0.0843	2.91
map_reduce	3.89	1	83.0%	18.8 s	22 s	23.4 s	$0.0819	0.89
map_reduce	3.89	2	83.5%	18.9 s	21.2 s	22.9 s	$0.081	1.79
map_reduce	3.89	3	84.0%	19.5 s	53.1 s	1 min 7 s	$0.0808	2.70
map_reduce	3.89	4	83.5%	4 min 40 s	7 min 14 s	8 min 9 s	$0.0814	3.21
map_reduce	3.89	6	82.5%	19 min 38 s	27 min 45 s	28 min 24 s	$0.0824	3.13
map_reduce	3.89	8	81.5%	29 min 46 s	34 min 1 s	36 min 59 s	$0.0834	3.03
debate	1.95	1	90.5%	28.1 s	46.6 s	56.5 s	$0.13	0.97
debate	1.95	2	90.0%	10 min 34 s	15 min 12 s	16 min 23 s	$0.13	1.72
debate	1.95	3	90.0%	36 min 19 s	51 min 41 s	55 min 44 s	$0.13	1.71
debate	1.95	4	90.5%	54 min 9 s	67 min 37 s	74 min 38 s	$0.13	1.73
debate	1.95	6	92.0%	73 min 38 s	81 min 57 s	83 min 38 s	$0.12	1.74
debate	1.95	8	92.5%	81 min 3 s	88 min 4 s	89 min 23 s	$0.12	1.76

The chooser's tables (chapter 9)

1,000 simulated workflows per row, 1,500-token tool results; arrivals 1 a minute for 4 sub-questions and 0.25 for 16. The closed form is chapter 5's success times the chance every call gets through its retries.

sub-questions	pattern	success (95% interval)	closed form	p95	$ per success	capacity / min
4	single	85.0% (82.7%–87.1%)	85.7%	19.9 s	$0.0522	4.83
4	supervisor	82.4% (79.9%–84.6%)	81.7%	44.8 s	$0.0999	2.92
4	hierarchical	78.8% (76.2%–81.2%)	77.7%	57.2 s	$0.13	2.43
4	swarm	76.9% (74.2%–79.4%)	76.4%	1 min 40 s	$0.15	1.77
4	debate	87.3% (85.1%–89.2%)	87.7%	13 min 5 s	$0.19	1.23
4	map_reduce	83.4% (81.0%–85.6%)	82.9%	22.5 s	$0.0959	2.97
16	single	10.5% (8.7%–12.6%)	11.2%	51.7 s	$1.74	1.48
16	supervisor	47.2% (44.1%–50.3%)	46.4%	3 min 12 s	$0.63	0.82
16	hierarchical	45.2% (42.1%–48.3%)	45.0%	3 min 37 s	$0.69	0.78
16	swarm	0.8% (0.4%–1.6%)	0.7%	3648 min 10 s	$145.15	0.15
16	debate	13.1% (11.1%–15.3%)	12.3%	16 min 33 s	$3.98	0.37
16	map_reduce	52.3% (49.2%–55.4%)	50.5%	2 min 32 s	$0.53	0.84