Choosing a pattern

A decision guide built from the simulation: set constraints on success, latency, cost and throughput, and watch them strike out patterns until one is left.

Loading the animation…

Concept

Advice about agent architectures is plentiful and mostly unfalsifiable. A simulation turns it into a table you can argue with. Every pattern of chapter 5 is run through chapter 8's discrete-event simulation on the same task and load (1,000 workflows each, with the rate limits, failures and retries of chapter 8), and each row records what you would measure in production: the share of workflows that succeed (with its 95% interval), the p95 latency, the cost per successful task, and the capacity the rate limits allow. Choose your constraints; the animation applies them one at a time, strikes out the patterns that miss each, and picks the most reliable survivor, the cheaper one on a tie.

What the two tasks say:

Four sub-questions (arriving 1 a minute): the single agent is the answer to almost every question you can ask. It succeeds 85.0% of the time, has the lowest p95 (19.9 s), the lowest cost per success ($0.0522) and the most headroom (4.83 workflows a minute). Debate is a little more reliable (87.3%), so "success at least 80%" with no other constraint picks it, but at this load it is near its capacity (1.23) and its p95 is 13 min 5 s. Add any latency constraint and the single agent is back.
Sixteen sub-questions (0.25 a minute): nothing reaches 80%, so the first lesson is that the task needs a better model or a verifier (chapter 7), not a cleverer topology. Ask for 40% within three minutes and only map-reduce is left: 52.3%, p95 2 min 32 s, $0.53 per success. The supervisor is about as reliable (47.2%) but slower (3 min 12 s); the single agent, starved by its own context, succeeds 10.5% of the time; the swarm, whose shared history outgrows the input limit, can barely run at all (0.15 workflows a minute).

Three rules come out of the runs rather than from opinion:

Start with one agent, and split only when its context grows past what the model handles well. The split pays when it shortens contexts; it costs when it only adds calls.
Parallelise to cut latency. Map-reduce costs about what a supervisor does (it drops the routing calls, but its reducer reads every answer at once) and is several times faster (25.6 s against 2 min 2 s uncontended on the large task), until the rate limit binds (chapter 6).
Check capacity before latency. A pattern within its latency budget at one workflow a minute can be hours behind at four, and the binding limit is usually tokens, not concurrency (chapter 8).

Maths

Each row is one seeded run of the simulation. Success is a binomial proportion over 1,000 workflows, shown with its 95% Wilson interval; the closed form of chapter 5 multiplied by the chance that every call gets through its retries, $\prod_i (1 - f^{A})$ with $f$ = 0.03 and $A$ = 3, agrees with it (the map-reduce row: 50.5% closed form, 52.3% simulated). Cost per success is the total cost of all workflows, failed ones included, divided by the successes, so an unreliable pattern pays for its failures. Capacity is $\mu$ of chapter 8.

The choice is a constrained maximisation: filter by the constraints, then maximise success, breaking ties by cost. With noisy estimates, two patterns whose intervals overlap are not distinguishable, and the honest answer is "either, and measure".

Code

The chooser, cut from the vendored engine (study.ts):

for (const r of table) {
  if (!alive.includes(r.pattern)) continue;
  const v = r[c.key];
  const good = c.op === ">=" ? v >= lim : v <= lim;
  (good ? keep : out).push(r.pattern);
}