Question 1

Why is batch-1 decode on a 70B model bandwidth-bound, and what is its latency floor on two H100s?

Accepted Answer

Decode reads every weight once per token. The answer turns that sentence into a millisecond floor, then accounts for the three things that keep a real server above it.

Question 2

Derive the FLOPs per token for a dense transformer from its config. Where does the 2N rule stop being accurate?

Accepted Answer

Two FLOPs per parameter is the number everyone quotes. The answer interviewers want rebuilds it from the config, finds the parameters that do no arithmetic, and names the context length where the rule breaks.

Question 3

Why does prefill turn compute-bound after a few hundred tokens on an H100, and how long does a 4K-token prompt take on a 70B model?

Accepted Answer

Decode and prefill run the same weights through the same kernels, yet one waits on memory and the other on tensor cores. The answer finds the prompt length where they trade places and turns it into a TTFT.

Question 4

Decode attention intensity is fixed at the query-to-KV head ratio. Derive that, then tell me what MLA does to it.

Accepted Answer

Batching rescues the weight GEMMs at decode but never the attention kernel, and one head-count ratio explains why. The answer derives it, then shows how MLA moves attention to within a hair of the H100 ridge.

Question 5

A vendor claims 1,000 tokens per second per user for a 70B model on one H200. Sanity-check that from the roofline.

Accepted Answer

One division tells you whether a headline number is physics or marketing. The answer runs it, then works out which unstated condition would make the claim true and what to ask the vendor for.

Question 6

We are moving a 70B model from H100s to B200s. How do decode latency and throughput change, regime by regime?

Accepted Answer

The datasheet ratios are close to each other, so the lazy answer is one speedup for everything. The answer splits it into three regimes, and the biggest win turns out to come from capacity, not bandwidth.

Question 7

DeepSeek V3 has 671B parameters but only 37B active. Why does a decode step at batch 64 read far more than 37B?

Accepted Answer

Active parameters describe one token. A decode step serves a batch, and the batch decides how many experts get pulled through HBM. The answer computes that number and shows where the MoE cost advantage actually comes from.

Question 8

How would you measure MBU on a running server? What do you time, what do you count, and which numbers make the result wrong?

Accepted Answer

MBU is one division, and that is why it is so easy to get wrong. The answer defines both operands precisely on a real server and then shows six ways a dashboard can report a number 30 points off.

Question 9

Your decode step runs at 60% of the bandwidth floor. Where does the rest of the time go? Build me the budget.

Accepted Answer

A 60% MBU number is a symptom, not a diagnosis. The answer rebuilds the floor from bytes, then splits the missing 40% into buckets a profiler can confirm and a team can own.

Question 10

Long context: at what sequence length do KV reads overtake weight reads for a 70B GQA model at batch 16?

Accepted Answer

The crossover falls out of one division, and the number it gives lines up exactly with something else on the GPU. Spotting that coincidence is what the interviewer is waiting for.

Question 11

Derive the curve of decode tokens per second against batch size, and show me where the knee is.

Accepted Answer

Step time is a straight line in batch size, so throughput is a hyperbola with a ceiling. The answer finds the knee in closed form and shows why the ceiling usually comes from the KV cache, not the tensor cores.

Question 12

Estimate the energy per output token for a 70B model on H100s from power and throughput. What moves the number most?

Accepted Answer

Joules per token is power over throughput, and the power term is the one candidates obsess over. The answer shows why the denominator moves the result by about 50x while the numerator is pinned under a datasheet cap.

Question 13

An 8B model at batch 1 on an H100 should take 4.8 ms per step but takes 12 ms. Name the three likeliest causes, with numbers.

Accepted Answer

A 2.5x gap at batch 1 almost never comes from the GPU being slow. The answer sorts the suspects by what a single profiler trace shows and puts a millisecond figure on each.

Question 14

Rank the decode-time arithmetic intensity of the QKV projection, attention, the MLP and the LM head. Which ones can fusion help?

Accepted Answer

Every weight GEMM ties, attention sits at a constant the batch cannot move, and the ranking flips at batch 8. The answer then separates what fusion can fix from what only the model architecture or the batch can.

Question 15

Design a roofline-based performance model that predicts TTFT and TPOT for any model, GPU and batch. Where will it be wrong?

Accepted Answer

Twenty lines of roofline arithmetic get within striking distance of a real server, and the interviewer knows it. What they are scoring is the list of places the model breaks, and how you would catch each one before a capacity plan rests on it.