Inference physics chains: the ridge, the decode floor and what a step costs

These are the numbers a physics round asks for before anyone draws a box. Every hardware figure is a dense peak from the part's datasheet, as recorded in the site's hardware table; sparse peaks are twice as large and do not apply to a dense model.

The ridge point

inputs   dense BF16 peak and HBM bandwidth per GPU
H100 SXM 989 TFLOP/s / 3.35 TB/s = 295 FLOP per byte
H200 SXM 989 TFLOP/s / 4.8 TB/s  = 206 FLOP per byte
B200     2,250 TFLOP/s / 8 TB/s  = 281 FLOP per byte
H100 FP8 1,979 TFLOP/s / 3.35 TB/s = 591 FLOP per byte
sanity   batch-1 decode does about 1 FLOP per byte: bandwidth-bound by
         two orders of magnitude on all three parts

Using the "with sparsity" peak doubles the ridge and moves every crossover after it. The H200 has the same compute as the H100 and more bandwidth, so its ridge is lower: more work becomes compute-bound.

Drilled in: The roofline model for inference

The batch-1 decode floor

inputs   Llama 3.1 8B, BF16: 16.06 GB of weights read once per step
formula  step >= bytes read / bandwidth
H100     16.06 GB / 3.35 TB/s = 4.79 ms   (209 tokens/s)
H200     16.06 GB / 4.8 TB/s  = 3.35 ms   (299 tokens/s)
B200     16.06 GB / 8 TB/s    = 2.01 ms   (498 tokens/s)
sanity   the floor scales with bandwidth alone; the B200's extra compute
         does nothing for this step

A floor is a lower bound. It leaves out KV reads, kernel launches and sampling, so a real step sits above it, and how far above is what MBU measures.

Drilled in: Batch-1 decode floor on two H100s · H100 to B200 by regime

Prefill time

inputs   Llama 3.3 70B, 70.55e9 parameters, BF16, TP=2 on H100 SXM
         a 4,096-token prompt; dense FLOPs about 2 x parameters per token
linear   2 x 70.55e9 x 4,096 = 578 TFLOP
attention 2 x 80 layers x 4,096^2 x 8,192 = 22 TFLOP (3.7%)
time     600e12 / (2 x 989e12) = 303 ms at peak
sanity   0.074 ms per prompt token; real prefill runs below peak, so
         divide by the MFU you can defend before quoting a TTFT

The 303 ms is a floor at 100% of peak. Quote it with the MFU assumption attached, because the interviewer will ask for one.

Drilled in: FLOPs per token from the config · Prefill time for a 4K prompt on 70B

Where batching crosses the ridge

inputs   one weight matrix of N parameters in BF16 (2N bytes)
decode   batch B: 2NB FLOPs over 2N bytes = B FLOP per byte
prefill  chunk of T tokens: 2NT FLOPs over 2N bytes = T FLOP per byte
H100     compute-bound once B or T passes about 295
FP8 W8A8 2NT FLOPs over N bytes = 2T, against a 591 ridge: T = 295 again
sanity   halving the bytes and doubling the peak leaves the crossover
         where it was

The crossover covers the weight GEMMs only. Attention reads each sequence's own KV, which no other sequence shares, so long-context decode stays bandwidth-bound at any batch size.

Drilled in: Arithmetic intensity per decode op

When KV reads overtake weight reads

inputs   Llama 3.1 8B BF16: 16.06 GB of weights, 128 KiB of KV per token
         batch 64, every sequence at context L
formula  KV per step = B x L x 131,072 B; equal to weights when
         L = 16.06e9 / (64 x 131,072) = 1,914 tokens
sanity   past 1,914 tokens of context the KV, not the model, sets the
         step time at this batch

Bigger batches pull the crossover in: at batch 256 it is 479 tokens. Capacity plans that price decode by weights alone are wrong for any chat workload with history.

Drilled in: Context where KV reads overtake weights

A batched decode step

inputs   Llama 3.1 8B BF16 on one H100 SXM, batch 64, 2,048 tokens each
KV       64 x 2,048 x 131,072 B = 17.2 GB
step     (16.06 + 17.2) GB / 3.35 TB/s = 9.9 ms at the floor
output   64 / 9.9 ms = 6,450 tokens/s in aggregate, 101 per stream
sanity   batching multiplied throughput by 31 and cost each stream half its
         batch-1 speed

Throughput per GPU and speed per user move in opposite directions along the same curve. Which point on it a product wants is the decision the arithmetic serves.

Drilled in: Decode throughput-latency curve

One output token against one prompt token

inputs   the Llama 3.3 70B TP=2 deployment above, batch 1, 4,096-token context
decode   weights 70.55 GB + KV 0.67 GB per GPU = 71.2 GB / 3.35 TB/s = 21.3 ms
prefill  0.071 ms per prompt token for the linear layers
ratio    21.3 / 0.071 = 300x
sanity   the ratio is the ridge point, as it must be at batch 1

Providers price output above input for this reason, and by much less than 300x, because a loaded server spreads each decode step's weight read across many sequences.

Drilled in: Prefill versus decode

MBU from a measured step

inputs   suppose Llama 3.1 8B at batch 1 measures 6.0 ms per step on an H100 SXM
formula  MBU = bytes the step must read / step time / peak bandwidth
         16.06e9 B / 6.0e-3 s / 3.35e12 B/s = 80%
sanity   80% of the floor is good for batch-1 decode; the missing 20% is
         launches, sampling and the parts of the step that are not reads

Count only bytes the step must read: weights once, plus each sequence's KV. Adding activations or counting a shared prefix once per sequence inflates MBU past what the memory delivered.

Drilled in: Measure MBU on a running server · Model bandwidth utilization