Speculative Decoding Interview Questions

Three parts, and most candidates finish one. The algebra fits on a line, the proof fits in three, and the part interviewers remember is the batch size at which the whole trick runs out of spare compute.

Three knobs, one distribution, and an ordering question that most candidates have never thought about. The answer works one six-token example through every knob and shows the order changing which tokens can be sampled.

At temperature zero, verification is a prefix match, and the whole implementation fits in a dozen vectorized lines. Interviewers grade the three details around it: the bonus token, the KV slots to roll back, and why the output can still differ from plain greedy.

One option drafts four tokens for six percent of a target step but guesses each position blind; the other conditions every guess and costs nearly half a step. The answer prices both on Llama 3.3 70B and finds the batch size where the winner flips.

A draft tree turns a handful of guesses into many candidate continuations, and a mask lets one pass check them all. The answer builds the mask, counts what it costs, and shows where a ten-node tree stops beating a five-token chain.

A one-layer drafter that beats an 8B model on acceptance sounds wrong until you ask what it is reading. The answer prices the EAGLE head on Llama 3.3 70B, explains the feature trick, and names the three changes in EAGLE-3.

DeepSeek V3 ships a drafter it trained alongside the model for free. The answer walks the MTP module's wiring and loss, prices a draft step against a 37.6B-active target, and reproduces the reported 1.8x from first principles.

A drafter that costs nothing and needs no training sounds too good, and on open chat it does nothing. The answer traces a code-rename request step by step, counts accepted tokens under three match-length settings, and says when to turn it off.

Compute acceptance from counter snapshots, reject reset-contaminated windows, and handle EOS before inferring request-step counts. Then decide what a matched-load baseline must measure.

The model never sees the grammar; the sampler does, as a 16 KB bitmask per request per step. The answer builds that mask from an automaton, prices the naive version at ten milliseconds, and shows how precompilation removes almost all of it.

A per-request schema compile stalls the scheduler, a per-step mask must beat the forward pass, and skipping forced characters can push the model off its own tokenization. The answer handles all three with numbers from one realistic schema.

Four beams over a 1,000-token prompt cost 600 MiB if each keeps its own cache and as little as 156 MiB if they share blocks. The answer implements the fork, the copy-on-write and the prune, then shows what vLLM actually does today.

Lossless is a statement about one distribution, and quantization changes which one without raising any error. The answer separates the draft case from the target case, then shows two ways an FP8 deployment breaks exactness without touching the rejection rule.

Code wants eight draft tokens and chat wants two, and a fixed gamma serves both badly. The answer simulates three policies on a mixed workload, finds the gain is 5% with a cheap drafter and 19% with an expensive one, then explains why batching takes most of it away.

The same drafter that triples speed at 3 a.m. costs a third of peak throughput at 6 p.m. if left alone. The answer builds a step-time model for Llama 3.3 70B on eight H100s, derives a draft length per batch bucket, and shows why context length moves the whole table.