The crossover falls out of one division, and the number it gives lines up exactly with something else on the GPU. Spotting that coincidence is what the interviewer is waiting for.
an account raises the free count per topic
← Inference Physics & Roofline / 10
mediumNewAnthropicGoogle DeepMindOpenAI
Long context: at what sequence length do KV reads overtake weight reads for a 70B GQA model at batch 16?
The crossover falls out of one division, and the number it gives lines up exactly with something else on the GPU. Spotting that coincidence is what the interviewer is waiting for.
Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.
UP NEXT ON YOUR JOURNEY
Next in this trackDerive the curve of decode tokens per second against batch size, and show me where the knee is.Next in this trackEstimate the energy per output token for a 70B model on H100s from power and throughput. What moves the number most?Next in this trackAn 8B model at batch 1 on an H100 should take 4.8 ms per step but takes 12 ms. Name the three likeliest causes, with numbers.
DISCUSSION · 0
Nobody has commented on this answer. Add a correction or another way to work it.