Hard inference interview questions

Batching rescues the weight GEMMs at decode but never the attention kernel, and one head-count ratio explains why. The answer derives it, then shows how MLA moves attention to within a hair of the H100 ridge.

Active parameters describe one token. A decode step serves a batch, and the batch decides how many experts get pulled through HBM. The answer computes that number and shows where the MoE cost advantage actually comes from.

A 60% MBU number is a symptom, not a diagnosis. The answer rebuilds the floor from bytes, then splits the missing 40% into buckets a profiler can confirm and a team can own.

Step time is a straight line in batch size, so throughput is a hyperbola with a ceiling. The answer finds the knee in closed form and shows why the ceiling usually comes from the KV cache, not the tensor cores.

A 2.5x gap at batch 1 almost never comes from the GPU being slow. The answer sorts the suspects by what a single profiler trace shows and puts a millisecond figure on each.

FP8 halves the cache and the attention bytes, and on short prompts it looks free. The answer explains why the scale matters less than people think for rounding and more than they think for clipping, and why the failures hide at 128K.

Half the layers capped at a 128-token window sounds like a big saving at 128K. The arithmetic says the saving is set by the layer ratio, not the window, and the allocator decides whether you get it at all.

Recomputing a prefix costs FLOPs proportional to the model, fetching it costs bytes proportional to the cache. One ratio decides it, and the answer shows by how much fetching wins and the three situations where it still loses.

All three policies keep the server alive. They differ in who waits, who redoes work, and which percentile absorbs the damage, and the numbers for a 6K-token victim make the choice for you.

The transfer looks like gigabytes per request until you divide by prefill time. Then the prompt length cancels, and the real question becomes which models and which GPUs break the link.

A fixed recurrent state still costs memory for every live request and saved checkpoint. Work through a state inventory, then find a restart boundary that every layer can actually restore.

A priority flag fixes the queue and leaves the real damage in place: batch sequences that are already running slow every interactive token. The answer puts a number on the KV the batch class may hold.

The 70% rule of thumb falls out of one formula once you plug in a prefill time and a p99 target. The answer derives it, shows how prompt-length variance moves it to 50%, and lists the four ways an LLM engine breaks the textbook queue.

Request-fair scheduling hands 96% of the GPU to the tenant with long documents. The answer derives the token weights from the roofline, shows the counter that enforces them per step, and the one rule that stops a tenant from banking credit.

Scheduling step n+1 before step n's tokens exist sounds impossible. The answer shows the placeholder trick that makes it work, what it costs at an end-of-sequence token, and the configurations where vLLM switches it off for you.

Accepting everything during a one-minute spike leaves the API serving dead requests for two minutes after it ends. The answer does the queue arithmetic, picks the rejection rule, and writes the response that keeps clients from making it worse.

Storing the attention matrix for backward costs gigabytes per layer; recomputing it costs extra FLOPs. The answer derives the gradient, shows the one scalar per row that makes recomputation exact, and the identity that removes the last N-squared term.

Batch-1 decode at long context gives the GPU eight thread blocks and 132 SMs. The answer counts the waves, picks the split count from a wave-efficiency rule rather than a guess, and derives the merge that makes the result exact.

Fusing RoPE saves a read and a write of Q and K, which is real but small in prefill. The precision bug is not small: one tensor in the wrong dtype scrambles positional information by a few hundred tokens. The answer sizes both.

An error that grows with context length is a running sum losing increments, and BF16 stops adding 1 to a total at 256. The answer turns the symptom into a bisection, finds the state held in the wrong dtype, and shows the error curve that proves it.

Review a shape-by-shape timing fixture, catch a ragged-cache correctness failure, and calculate whether selective dispatch pays for itself. Includes a small reference you can run without a GPU.

A matrix-vector product has one job: keep enough bytes in flight. The answer counts how many that is per SM, then finds the three places a decode GEMV quietly drops below it.

Every dimension of a GEMM tile is pinned by a different piece of hardware. The answer derives the reuse that sets the M and N sides, the swizzle row that sets K, and the register and shared-memory budgets that stop the tile growing.

A Python loop over 256 experts is correct and slow for three separate reasons, only one of which is launch count. The answer builds the expert-sorted layout the fused kernel runs on and counts what it costs in padding.

A 4-bit weight is only cheaper if turning it back into FP16 costs less than the bytes it saved. The answer counts the instruction budget per weight at full HBM bandwidth and shows the bit trick that makes the conversion fit.

CUDA lets you reserve part of L2 for data you choose, and the KV cache is re-read every step. The answer measures what fits against what a decode step streams, finds the one case where pinning pays, and the kernel change that pays more.

Weight-only INT4 shrinks the bytes but keeps the arithmetic in BF16. The answer finds the batch size where that bargain flips, which does not depend on the model, then explains why a code benchmark notices before perplexity does.

A handful of channels with huge activations wreck the INT8 grid for every other channel in the token. The answer shows why the obvious fix cannot run on a tensor core, what SmoothQuant does instead, and why FP8 makes most of the problem disappear.

A few matrices cause most of the damage when a model is quantized, and keeping them wide costs little memory. The answer builds the sweep that finds them, prices it in GPU hours, and explains why the sum of single-layer damages is not the damage of the mix.

Reduction order explains the last-bit drift, but a quantized server has a second, much larger leak: a scale computed over the whole batch lets one user's activations change another user's logits. The answer separates the two and fixes both.

QAT never makes a model faster than PTQ at the same format; both produce identical tensors for the kernel. The answer reframes what QAT buys, prices a QAT run in GPU hours against the fleet it could shrink, and names the cases where a serving team should not touch it.

Three parts, and most candidates finish one. The algebra fits on a line, the proof fits in three, and the part interviewers remember is the batch size at which the whole trick runs out of spare compute.

A draft tree turns a handful of guesses into many candidate continuations, and a mask lets one pass check them all. The answer builds the mask, counts what it costs, and shows where a ten-node tree stops beating a five-token chain.

DeepSeek V3 ships a drafter it trained alongside the model for free. The answer walks the MTP module's wiring and loss, prices a draft step against a 37.6B-active target, and reproduces the reported 1.8x from first principles.

The model never sees the grammar; the sampler does, as a 16 KB bitmask per request per step. The answer builds that mask from an automaton, prices the naive version at ten milliseconds, and shows how precompilation removes almost all of it.

A per-request schema compile stalls the scheduler, a per-step mask must beat the forward pass, and skipping forced characters can push the model off its own tokenization. The answer handles all three with numbers from one realistic schema.

Tensor parallelism divides the weight bytes but not the all-reduce count, so each doubling buys less than the one before. The answer prices that fixed cost, then finds the workload where TP=8 wins anyway.

Every MoE layer sends each token out to its experts and gathers the results back, twice per layer, 58 times per step. The answer turns DeepSeek V3's config into bytes per token, splits them between NVLink and the NIC, and shows what node-aware dispatch saves.

An 8k-token prompt on a 70B model leaves 2.7 GB of KV that has to cross the network before decode can start. The answer shows that the sustained bandwidth is modest, that the latency is the real problem, and which side of the link runs out first.

A ring all-reduce is bandwidth-optimal and takes fourteen synchronized steps on eight GPUs. At 16 KB per call, the steps are the cost. The answer builds the latency-bandwidth model, finds the crossover size, and checks it against the threshold vLLM ships.

Same eight GPUs, same weights per GPU, same KV capacity. One layout pays 252 all-reduces per token, the other pays for every layer twice in wall-clock time. The answer works both, then finds the precision choice that forces the second.

Tensor parallelism across nodes would all-reduce every activation over the network. Splitting the sequence instead sends only the KV, which for a GQA model is small. The answer sizes that traffic, finds when it hides behind attention, and fixes the causal-mask imbalance that naive splitting creates.

Inside one batch there is nothing to overlap: every layer waits for its all-to-all. Split the batch in two and one half computes while the other communicates. The answer models the gain, finds the batch size where splitting makes things slower, and checks it against a shipped default.

Both reuse KV for shared prefixes, and on a 2K-token chat prompt their hit rates differ by under one percent. The answer finds the differences that do matter, from eviction policy to whether a cache key means anything outside one process.

The GPU finishes a decode step and waits while Python appends tokens, checks stop strings and plans the next batch. The answer shows how the loop reorders that work, where the placeholder tokens live, and the request that pays for it.

Eight tenants, eight adapters, one base model and one batch. The answer explains why the extra cost scales with distinct adapters rather than requests, and which default serializes them.

About a hundred lines of Python that admit, chunk, preempt, sample and stream. The interviewer is watching where the KV accounting lives and what happens when the pool runs dry.

Compare three capture policies using a memory ledger and a matched workload. Recover the missing KV pages, then defend a policy against both capacity and latency constraints.

Requests-per-minute limits let one tenant with 128K-token prompts take a fifth of a replica's memory per request. The answer prices quotas in the resource tokens actually consume, then isolates the three things tenants share: compute, KV memory and the prefix cache.

A chat fleet sized for its peak sits more than half idle over a day, and a batch API can soak it up. The answer measures the idle replica-hours, finds what limits batch throughput (it is not compute), and decides where the two workloads meet.

Five hundred merged fine-tunes need five hundred GPUs before the first request. Served as adapters on a shared base they need a few dozen. The answer sizes the adapters, places them in a three-tier memory hierarchy, and finds the one setting that silently multiplies their memory by eight.

Autoscaling cannot save a failover when a replica takes six minutes to come up, so the spare capacity has to exist before the region dies. The answer solves for the smallest fleet that survives any single region loss, and places the spare next to the region most likely to need it.

Offline work changes the regime: the batch is as big as memory allows, latency is irrelevant, and output tokens stop being expensive. The answer prices both passes in GPU-hours, finds which one is the bill, and cuts it.

Inspect tokenized prompts and cache-counter deltas to diagnose a TTFT regression. A runnable prefix-boundary check and a routing follow-up test whether the evidence actually supports the diagnosis.

FP8 halves the weight bytes, and the dashboard says the replica got slower. The answer computes what FP8 should have bought, splits the gap into kernel, scale and config terms with one profiler trace, and catches the regression that is not in the GPU at all.

Perplexity matches the BF16 baseline to two decimals and the needle-in-a-haystack score collapses past 16K. The answer explains why the two checks measure different things, isolates weights, KV and config with a small grid, and shows a measured case of a quantization scheme that only breaks at length.

Both numbers can be true at once. The answer shows how to turn the gap into a short list of measurable dilutions, which one usually eats most of the 2x, and the replay test that settles it.

A regular period is a clue most candidates throw away. The answer turns it into a ranked list of periodic sources, shows why a few slow steps out of hundreds own the p99, and names the one correlation that separates the top two.

A hang gives you no error, no log line and no metric. The answer shows how to make every rank say where it is stuck, why a wrong network transport makes things slow but rarely hangs them, and the order that finds the cause in minutes.

Small models make the GPU step so short that host work stops hiding behind it. The answer shows how to prove that with a timeline instead of a hunch, how much throughput a few milliseconds of Python cost, and which fixes help in which order.

The KV pool is preallocated, so the growth is somewhere the engine does not budget. The answer splits memory into what is fixed and what can grow, turns a growth rate into bytes per request, and uses that one number to rule out half the suspects.

Use cold-schema and warm-batch traces to separate compilation from recurring stalls. Identify the exposed delay, choose a targeted experiment, and preserve output correctness.

Compare expert counts with rank-level completion times. Explain why the longest combine span can occur on a lightly loaded rank, then test a placement change against a communication control.

Reconstruct token arrival from a request-correlated trace. Separate transfer overlap from decode admission, and show how first-token delivery changes the metric that exposes a handoff stall.

Unused logical KV pages can coexist with exhausted device memory. Reconcile expanded image tokens, retained embeddings and phase-specific peaks before changing a decoder token limit.

Replay the same burst through affinity, round-robin and predicted-finish routing. Then remove local reservations or delay an eviction notice and explain exactly why the decisions change.

An event can contain metadata, one token or several tokens. Audit a buffered stream without inventing token timestamps, and separate client launch delay from the engine's execution time.

Forty lines of Python, and graders run property tests against them. The answer gets the three invariants right first, then the one copy-on-write case most implementations get backwards.

The accept rule is one line. The residual distribution, the bonus token, rows that drafted fewer tokens and a test that can tell a lossless verifier from a subtly biased one are the rest of the grade.

Matching a prefix is a trie walk. The grade is in the edge split, which cached tokens a running request pins, and an eviction loop that turns parents into leaves without ever freeing a slot twice.

Beam search reshuffles which hypotheses survive every step, and a naive cache copies every survivor's history. The answer forks block tables instead, gets the release order right so the last sharer writes in place, and proves it against a beam search that never pages.

Two prompts with the same 16 tokens in block 3 must not share block 3 unless blocks 0 to 2 match too. The answer chains the hashes, keeps freed blocks hittable until memory is needed, evicts tails before heads, and proves a colliding hash cannot hand a request the wrong KV.

A runnable allocator hides an exception-safety defect and a payload-aliasing defect. Reduce each failure to a short operation sequence, then test ownership and content against separate oracles.

A disconnect arrives after launch, then the request ID is reused. Implement deferred reclamation and prove that late or duplicate completions cannot emit stale output or free another request's pages.

The tensor has the right shape and every token ID is valid, yet one request receives another request's continuation. Reproduce the routing error and repair the launch-to-completion contract.

Accepted, emitted and computed tokens are different counts. Trace rejection, full acceptance, EOS and an output cap to find the state that the next forward pass is allowed to read.

A fixed global seed reproduces a launch order, not a request. Build a random-draw address that survives compaction, extra draft work and continuation, then state what it cannot guarantee.

One slow connection can stall unrelated generations or retain an unbounded output queue. Implement a byte budget that includes the pending write, then connect overflow and disconnects to engine cancellation.