Running at scale

Hundreds of workflows sharing one rate-limited endpoint: queueing, tail latency, retries and cost per successful task, and the knee where load meets capacity.

Loading the animation…

Concept

One workflow at a time tells you its latency and cost. A service runs hundreds at once, and then the questions change: how long do requests wait, how bad is the tail, and what does a successful task cost once retries and failures are counted? The animation is a discrete-event simulation, the same method used for the inference servers on the Inference Trade-offs site: 200 workflows arrive at random (a Poisson process), every model call joins one queue in front of the endpoint, the client admits calls while the concurrency limit (16) and the token buckets allow, and a failed call (3% of attempts) is retried after a jittered exponential backoff, up to 3 attempts.

Each pattern has a capacity: the workflows per minute the binding limit allows. A supervisor workflow makes 13 calls and sends 10,520 uncached input tokens, so at 40,000 input tokens a minute it can sustain 3.80 workflows a minute, and its requests (3.85) bind almost as soon. Below capacity, latency is just the workflow's own: at 2 a minute the supervisor's p95 is 43.4 s. At 3 a minute (load 0.79) calls start to queue and the p95 is already 1 min 6 s; at 4 a minute, past capacity, the queue grows for as long as arrivals continue and the p95 is 10 min 45 s. That is the knee: well below capacity, adding load costs almost nothing; near it, the tail explodes. The single agent's capacity is 7.87 a minute, so it sails through every load up to 6 a minute with a p95 near 19.1 s.

The cost per success barely moves with load ($0.0896 to $0.0828 for the supervisor): queueing costs time, not tokens. What it does cost is utilisation of the wrong thing: the concurrency slots are mostly idle (12% busy at 3 a minute) while the input-token bucket is the bottleneck. Raising the concurrency limit would change nothing; caching more of each prompt (cache reads do not count toward the input limit) or a higher tier would.

Maths

Let a workflow make $k$ calls with $I$ uncached input and $O$ output tokens in all. The buckets admit at most RPM $/k$ , ITPM $/I$ and OTPM $/O$ workflows a minute, and $C$ concurrent slots admit at most $C$ divided by the busy time per workflow; the capacity $\mu$ is the smallest. When the arrival rate $\lambda$ exceeds $\mu$ the queue grows without bound (the simulation's run ends only because arrivals stop); below it the queue is finite, but its waits grow sharply as $\rho = \lambda/\mu \to 1$ , the shape every queueing model shares (for the simplest, M/M/1, the mean time in the system is $1/(\mu - \lambda)$ ). The engine's tests check that the simulated throughput saturates at $\mu$ when it is overloaded.

Little's law, $L = \lambda W$ (J. D. C. Little, A proof for the queuing formula L = λW, Operations Research 1961, doi:10.1287/opre.9.3.383), holds for any stable system: the average number of workflows in the system is the arrival rate times the average time each spends there. In the simulation it is an identity, which the engine's tests check: the area under the "in system" curve equals the sum of the workflows' latencies.

Percentiles are nearest-rank: the $p_{95}$ of $m$ latencies is the $\lceil 0.95\,m \rceil$ -th smallest.

Code

The retry with full jitter, cut from the vendored engine (des.ts):

if (a < P.max_attempts) {
  let cap = P.backoff_ms * pow2(a - 1);
  if (cap > P.backoff_cap_ms) cap = P.backoff_cap_ms;
  push(at + rng.random() * cap, "retry", w, cid);
} else finishWf(at, w, "exhausted");