Practice tests · 30 items

Speculative & Constrained Decoding practice test

The draft and verify algebra with its losslessness proof, acceptance rate versus draft length, Medusa, EAGLE and multi-token prediction heads, tree verification, why speculation loses at high batch, sampling kernels, and grammar-constrained output. The pool for this topic holds 10 easy, 12 medium and 8 hard items. Each answer comes with its reasoning and a link to the question or concept page it was drawn from.

Configure a test

Topic

Difficulty

Questions

10 in this pool · about 7 min

Reveal answers

Sample items with answers

easy · sample

Batch-1 decode on a 70B model leaves an H100's tensor cores more than 99% idle. Why does speculative decoding turn that idle compute into speed?

Drafted tokens skip the target's attention layers, so each accepted token avoids most of the target's FLOPs

Accepted tokens arrive with KV entries the draft already computed, so the target never writes their cache rows

✓Checking several drafted positions in one target pass reads the weights once, the bytes of a single token

The target verifies at lower precision than it decodes, which doubles tensor-core throughput for those passes

Decode is bandwidth-bound. At batch 1 a step does about 1 FLOP per weight byte, against an H100 SXM ridge of 989 TFLOP/s over 3.35 TB/s, 295 FLOP/byte. Feeding the target gamma + 1 positions multiplies the FLOPs by gamma + 1 while the weight bytes barely change, so the pass costs roughly what one token cost. The target still runs every layer on every drafted token and writes its own KV for them, at its normal precision. The speedup is idle arithmetic converted into accepted tokens.

easy · sample

With draft length 4, the target accepts all four drafted tokens in one speculative step. How many tokens does that step emit?

4: the accepted drafts, since the target only checks the tokens it was handed

✓5: the four drafts plus a bonus token sampled from the target's last position

4, because one draft slot is spent as the anchor the target verifies against

8: the four drafts plus the four target predictions made at those same positions

The verification pass computes the target's distribution at gamma + 1 positions: after the last committed token and after each draft. When all gamma drafts pass, the distribution at the final position has not been used yet, so the engine samples one more token from it at no extra cost. That is why a step emits between 1 and gamma + 1 tokens. Counting 8 double counts: the target's prediction at an accepted position is that same token, not an additional one.

After the test

Picking an option is easier than deriving the answer at a whiteboard. These pages make you derive it: