disaggregation

The transfer looks like gigabytes per request until you divide by prefill time. Then the prompt length cancels, and the real question becomes which models and which GPUs break the link.

An 8k-token prompt on a 70B model leaves 2.7 GB of KV that has to cross the network before decode can start. The answer shows that the sustained bandwidth is modest, that the latency is the real problem, and which side of the link runs out first.

Eight GPUs per NVLink domain shaped every serving layout of the Hopper era. Seventy-two moves the boundary that expert parallelism and KV transfer used to cross over the NIC. The answer prices what moves inside the rack, what still does not fit, and the failure mode that comes with it.

A 1 TB FP8 model fits on one B200 node and needs four H100 nodes. That single fact sorts the two generations into roles. The answer assigns them, sizes both pools, and then deals with the precision mismatch that shows up when the two halves disagree about the weights.

Most designs draw the router first. The answer that passes does the arithmetic first: tokens per second in each phase, replica-seconds per second, a utilization ceiling, then dollars, and finds that the prompt, not the reply, is two thirds of the bill.

A cold 32K prefill on a 70B model barely fits a second of compute on eight H100s, before any queueing. The design that passes makes almost every request warm, then sizes two pools from what remains.

Reconstruct token arrival from a request-correlated trace. Separate transfer overlap from decode admission, and show how first-token delivery changes the metric that exposes a handoff stall.