Find your next
hard question.
Derive the number. Explain the mechanism. Defend the decision. Every worked answer includes follow-ups and a scoring rubric.
- 01
Why is batch-1 decode on a 70B model bandwidth-bound, and what is its latency floor on two H100s?
Decode reads every weight once per token. The answer turns that sentence into a millisecond floor, then accounts for the three things that keep a real server above it.
Answer available - 02
Derive the FLOPs per token for a dense transformer from its config. Where does the 2N rule stop being accurate?
Two FLOPs per parameter is the number everyone quotes. The answer interviewers want rebuilds it from the config, finds the parameters that do no arithmetic, and names the context length where the rule breaks.
Answer available - 03
Why does prefill turn compute-bound after a few hundred tokens on an H100, and how long does a 4K-token prompt take on a 70B model?
Decode and prefill run the same weights through the same kernels, yet one waits on memory and the other on tensor cores. The answer finds the prompt length where they trade places and turns it into a TTFT.
Answer available - 04
Decode attention intensity is fixed at the query-to-KV head ratio. Derive that, then tell me what MLA does to it.
Batching rescues the weight GEMMs at decode but never the attention kernel, and one head-count ratio explains why. The answer derives it, then shows how MLA moves attention to within a hair of the H100 ridge.
Answer available - 05
A vendor claims 1,000 tokens per second per user for a 70B model on one H200. Sanity-check that from the roofline.
One division tells you whether a headline number is physics or marketing. The answer runs it, then works out which unstated condition would make the claim true and what to ask the vendor for.
Answer available - 06
We are moving a 70B model from H100s to B200s. How do decode latency and throughput change, regime by regime?
The datasheet ratios are close to each other, so the lazy answer is one speedup for everything. The answer splits it into three regimes, and the biggest win turns out to come from capacity, not bandwidth.
Free with an account - 07
DeepSeek V3 has 671B parameters but only 37B active. Why does a decode step at batch 64 read far more than 37B?
Active parameters describe one token. A decode step serves a batch, and the batch decides how many experts get pulled through HBM. The answer computes that number and shows where the MoE cost advantage actually comes from.
Free with an account - 08
How would you measure MBU on a running server? What do you time, what do you count, and which numbers make the result wrong?
MBU is one division, and that is why it is so easy to get wrong. The answer defines both operands precisely on a real server and then shows six ways a dashboard can report a number 30 points off.
Free with an account - 09
Your decode step runs at 60% of the bandwidth floor. Where does the rest of the time go? Build me the budget.
A 60% MBU number is a symptom, not a diagnosis. The answer rebuilds the floor from bytes, then splits the missing 40% into buckets a profiler can confirm and a team can own.
Free with an account - 10
Long context: at what sequence length do KV reads overtake weight reads for a 70B GQA model at batch 16?
The crossover falls out of one division, and the number it gives lines up exactly with something else on the GPU. Spotting that coincidence is what the interviewer is waiting for.
Free with an account - 11
Derive the curve of decode tokens per second against batch size, and show me where the knee is.
Step time is a straight line in batch size, so throughput is a hyperbola with a ceiling. The answer finds the knee in closed form and shows why the ceiling usually comes from the KV cache, not the tensor cores.
Premium answer - 12
Estimate the energy per output token for a 70B model on H100s from power and throughput. What moves the number most?
Joules per token is power over throughput, and the power term is the one candidates obsess over. The answer shows why the denominator moves the result by about 50x while the numerator is pinned under a datasheet cap.
Premium answer - 13
An 8B model at batch 1 on an H100 should take 4.8 ms per step but takes 12 ms. Name the three likeliest causes, with numbers.
A 2.5x gap at batch 1 almost never comes from the GPU being slow. The answer sorts the suspects by what a single profiler trace shows and puts a millisecond figure on each.
Premium answer - 14
Rank the decode-time arithmetic intensity of the QKV projection, attention, the MLP and the LM head. Which ones can fusion help?
Every weight GEMM ties, attention sits at a constant the batch cannot move, and the ranking flips at batch 8. The answer then separates what fusion can fix from what only the model architecture or the batch can.
Premium answer - 15
Design a roofline-based performance model that predicts TTFT and TPOT for any model, GPU and batch. Where will it be wrong?
Twenty lines of roofline arithmetic get within striking distance of a real server, and the interviewer knows it. What they are scoring is the list of places the model breaks, and how you would catch each one before a capacity plan rests on it.
Premium answer - 01
Derive the KV cache footprint for MHA, GQA and MLA, then tell me how many 8K-token conversations fit on four H100s.
The formula takes one line. The answer interviewers score is the memory budget around it: what the engine reserves first, what tensor parallelism does to the heads, and why the count you get is a ceiling.
Answer available - 02
Why does a paged KV cache waste at most one block per sequence, and how would you pick the block size?
The bound is a one-line argument about where a sequence can be partly full. The block size is a three-way trade between that waste, kernel efficiency and how finely the prefix cache can match.
Answer available - 03
Account for every gigabyte on a GPU running vLLM: weights, KV pool, activations, CUDA graphs and NCCL buffers.
Weights and KV are the two numbers everyone knows. The answer that gets the offer accounts for the other few gigabytes, says which ones vLLM measures and which it guesses, and knows where an OOM comes from when the sum looks fine.
Answer available - 04
FP8 KV cache: what scales does it need, where is the error largest, and how would you validate it on long-context tasks?
FP8 halves the cache and the attention bytes, and on short prompts it looks free. The answer explains why the scale matters less than people think for rounding and more than they think for clipping, and why the failures hide at 128K.
Answer available - 05
Tensor parallelism beyond the KV head count replicates the cache. Derive the per-GPU KV at TP=16 for an 8-KV-head model.
Doubling the GPUs past the KV head count halves the weights per GPU and leaves the KV per GPU exactly where it was. The answer derives that, prices the duplicate copy, and picks the parallelism that does not pay it.
Answer available - 06
A model mixes sliding-window and global attention layers. Compute its KV footprint at 128K and tell me what the window actually saves.
Half the layers capped at a 128-token window sounds like a big saving at 128K. The arithmetic says the saving is set by the layer ratio, not the window, and the allocator decides whether you get it at all.
Free with an account - 07
Design KV offload to CPU memory. When does fetching blocks back over PCIe beat recomputing them?
Recomputing a prefix costs FLOPs proportional to the model, fetching it costs bytes proportional to the cache. One ratio decides it, and the answer shows by how much fetching wins and the three situations where it still loses.
Free with an account - 08
Prefix-cache eviction: LRU by block or by request? And what does a radix tree change about it?
A prefix cache only hits from the first token forward, so evicting the wrong block can strand gigabytes that stay resident and can never be hit. The answer derives how much, and which eviction order guarantees it never happens.
Free with an account - 09
Why can MLA decode attend over the 512-dimensional latent without decompressing keys, and what does absorption cost in FLOPs?
Absorption is one line of matrix associativity. The part interviewers score is the bill: attention gets more expensive per cached token, and the kernel lands in a roofline regime GQA never visits.
Free with an account