TL;DR: Medusa's four heads on Llama 3.3 70B add 4.47B parameters, about 6% of a target step, and run in parallel; an 8B draft model costs 11.4% of a step per drafted token, 46% for four. Medusa loses on acceptance because head guesses position without seeing the tokens before it, and wins back the difference with tree verification, which is only free at small batch.
How to approach it
Put both options into the same speedup formula: tokens per step divided by the cost of a step, with draft cost measured in bytes read relative to the target. Then ask what decides the inputs: the batch size (trees spend verification compute), whether the team can train anything (Medusa needs head training, a draft model may already exist), and whether outputs must be exactly the target's distribution at temperature above zero.
A strong answer
What each one is. A draft model is a smaller model sharing the target's tokenizer, Llama 3.1 8B for Llama 3.3 70B (both use the 128,256-token Llama 3 vocabulary). It runs autoregressively times, so draft token sees drafts to . Medusa (arXiv 2401.10774) bolts extra heads onto the target's last hidden state. Head is one residual block followed by its own vocabulary projection, and it predicts the token positions ahead from the same hidden state. All heads run once, in parallel, at the end of the target step.
Cost per step. In the bandwidth-bound regime, cost is bytes read:
target Llama 3.3 70B, BF16: 70.55B params = 141.1 GB per step
Medusa head (d = 8,192, V = 128,256):
residual block d x d + bias = 67.1M, vocab projection d x V = 1,050.7M
4 heads = 4 x 1,117.8M = 4.47B params = 8.94 GB -> 8.94 / 141.1 = 6.3% of a step
draft Llama 3.1 8B, BF16: 8.03B params = 16.06 GB -> c = 16.06 / 141.1 = 0.114
gamma = 4 sequential steps: 4 x 0.114 = 0.455 of a target step
That 0.455 is optimistic. A small model hits its kernel-launch floor long before its bandwidth floor, so measured draft steps usually cost more than the byte ratio says, and running the draft on the target's TP group spreads an already small matrix across GPUs.
Acceptance. The draft model's guesses are conditioned; Medusa's are not. Head 3 must guess the third token without knowing what heads 1 and 2 guessed, so its acceptance falls off faster with depth. Illustrative per-position conditional acceptance (measure your own; these only show the shape):
per-position acceptance E[tokens] cost speedup
draft model, chain 0.80 0.78 0.76 0.74 3.25 1.455 2.23x
Medusa, top-1 chain 0.65 0.50 0.40 0.35 2.15 1.063 2.02x
Medusa, tree of top-k 0.80 0.65 0.50 0.40 2.68 1.063 2.52x
E = 1 + a1 + a1 a2 + a1 a2 a3 + a1 a2 a3 a4
Trees are Medusa's answer to blind guessing: take the top few candidates from each head, form a tree of combinations, and verify all branches in one pass with a tree attention mask. If any branch matches, the step advances along it. The price is verification width. A tree of 64 nodes is free at batch 1, but at batch 8 it puts 512 tokens through the weights, past the roughly 295 tokens an H100 decode step can absorb before turning compute-bound (989 TFLOP/s dense BF16 over 3.35 TB/s). The draft model's chain verifies only 5 tokens per request and stays free up to about batch 59.
Training. Medusa-1 fine-tunes the heads on a frozen backbone, so the target's outputs are untouched; Medusa-2 trains heads and backbone jointly for better acceptance, which changes the target and needs a careful recipe to keep quality. The authors report over 2.2x for Medusa-1 and 2.3x to 3.6x for Medusa-2 in the arXiv abstract. A draft model needs no training if one exists in the family, but a mismatched tokenizer rules it out. Medusa also introduced "typical acceptance" for sampled decoding, which accepts more drafts but does not preserve the target distribution; if exactness matters, use standard rejection sampling with the heads' probabilities.
The decision: Medusa when the deployment runs small batches, a training pipeline exists, and memory is tight (8.94 GB of heads against 16.06 GB of draft weights plus its own KV cache). A draft model when no training budget exists, or when batch sizes are large enough that trees stop being free. If the team can train something, the EAGLE family usually beats both, because it conditions on the target's features like Medusa and on previous drafts like a draft model.
What interviewers probe next
"The batch is 64." Shrink the Medusa tree to a handful of nodes or a chain, where its acceptance trails the draft model's; at this batch the 6% overhead matters less than the verification width.
"Why does each head have its own 1B-parameter vocabulary projection?" Each head predicts a different position, so the paper gives each its own projection, initialized from the target's LM head; sharing one saves 3.15B parameters at some cost to acceptance.
"Does vLLM serve Medusa?" Yes, as method medusa in v0.31.0, which falls back to Model Runner V1 and disables async scheduling.
How it's scored
Weak: says Medusa is "faster because it has no draft model" without accounting for its lower acceptance at deeper positions.
Hire: prices both drafters as a fraction of a target step from bytes read and explains why head cannot see earlier guesses.
Strong Hire: connects tree verification to the ridge point and finds the batch where it stops being free, separates Medusa-1 from Medusa-2 on whether the target changes, and flags typical acceptance as not lossless.
Common mistakes
- Counting Medusa heads as tiny residual blocks and forgetting the vocabulary projections, which are 94% of their parameters.
- Pricing the draft model at its byte ratio when its real step time is set by launch latency.
- Comparing a Medusa tree to a draft-model chain at batch 1 and quoting the result for a batch-64 service.
- Pairing a draft and target with different tokenizers and expecting token-level verification to work.