Build a cost ledger before choosing a device
Calculate compute and byte floors for each phase, reject devices that cannot hold the working set, then test the remaining candidates at the batch and context lengths you intend to serve.
20 MIN
By InferenceInterviews · Updated
Course lessons Lesson 4 of 24
TL;DR: Reject devices that cannot hold the live working set, then compare per-phase compute and byte floors at the intended batch and context; a peak specification alone cannot select a serving device.
Where you are. You have a reproducible request contract. Build the first resource ledger before running an expensive device comparison.
A fast device that cannot run the experiment
Consider a synthetic dense decoder with eight billion parameters stored in two bytes each. Its weight payload is 16 GB, using decimal units. A proposed device offers 12 GB of usable memory. A bandwidth calculation may predict a short decode step, but the weights alone do not fit. The candidate needs a different representation, partitioning or offload plan before its local bandwidth number is meaningful.
Keep feasibility ahead of speed. Include weights, persistent state and peak temporary allocations. On an integrated device, shared host/device memory changes the capacity and traffic model; it does not eliminate bandwidth contention. On a partitioned accelerator, use the assigned memory and compute resources, not the full physical device's peak figures.
Derive a lower bound for each phase
For a matrix-heavy dense transformer, 2P FLOPs per token is a useful first approximation to weight-matrix work when a multiply-add counts as two FLOPs. It is not the whole model: attention over context, vocabulary projection choices, normalization and other operations need their own terms when material. MoE requires counting activated expert work separately from resident parameters.
Our synthetic device has an assumed effective ceiling of 200 TFLOP/s for the selected two-byte matrix arithmetic and 2 TB/s for the relevant memory traffic. These are exercise inputs, not a device benchmark or vendor specification. A prompt of 1,000 tokens has approximately 2 × 8e9 × 1000 = 16e12 FLOPs of weight-matrix work, giving an 80 ms compute floor. Reading the 16 GB weights once gives an 8 ms byte floor.
For one decode token, the corresponding compute floor is 16e9 / 200e12 = 0.08 ms, while one full weight read still costs 8 ms. The phase distinction follows from reuse. It is conditional: sufficiently large decode batches can raise arithmetic intensity, and small prefills can remain dominated by memory or launch overhead.
These bars omit KV traffic and attention work deliberately. They illustrate why the same weight matrices encounter different limits. They are not predicted end-to-end latencies.
Add the terms that can reverse the result
A lower bound is at least the maximum of compute time and byte-transfer time when those resources can overlap. Adding the floors blindly assumes serialization; taking their maximum and calling it a measured prediction assumes perfect overlap and ignores other stages. Keep both assumptions visible.
At batch 64, the dense weight-matrix work becomes 64 × 16e9 FLOPs, or 5.12 ms at the exercise compute ceiling. If weights are read once for the batch, the 8 ms weight floor remains. But each sequence also reads its context state, and longer contexts can make KV traffic dominant. Kernel tiling, insufficient concurrency and reductions can prevent either ceiling from being reached.
Compare device candidates with the same precision path, batch distribution and context lengths. A sparse throughput specification cannot stand in for dense arithmetic. Bidirectional aggregate link bandwidth cannot be inserted directly into a one-way transfer equation. Also separate cold model loading from steady execution and measure host launch gaps when short kernels leave the device idle.
A useful experiment table has columns for predicted dominant term, measured phase latency and the residual the model did not explain. If the residual grows with batch padding, investigate the padded work. If it grows with context, revisit attention and KV. The ledger is valuable because it can be disproved.
Do this before moving on
Change only the assumed memory ceiling from 2 TB/s to 1 TB/s. Recompute the two weight floors and identify which phase's dominant term changes. Then increase the prompt to 4,000 tokens and explain why multiplying the original total latency by four may fail once attention work matters.
Use the decode economics experiment to move batch size. Record a condition under which the usual “prefill compute, decode bandwidth” shortcut would give the wrong diagnosis.
Go deeper
- Roofline model for inference derives intensity and the boundary between resource ceilings.
- FLOPs from model configuration replaces the parameter shortcut with an explicit layer calculation.
- GPU memory hierarchy explains why locality and bytes in flight affect achieved bandwidth.
- Prefill versus decode develops the phase-dependent reuse model behind the exercise.
Key takeaways
- Prove working-set feasibility before comparing speed ceilings.
- State the precision, traffic boundary and overlap assumption.
- Add attention and KV costs when context makes them material.
- Use measured residuals to revise the resource model.
Check yourself
Work it out before revealing the explanation. Include the units, assumptions and the condition that would change your answer.
1At 1 TB/s, what are the exercise weight-read floors?
2At batch 64, what is the weight-matrix compute floor?
3A device has faster bandwidth but insufficient memory for weights. Is it a valid local-resident candidate?
4Why is max(compute floor, byte floor) not a latency prediction?
Sign in to track which lessons you have finished.