Skip to content
← Speculative & Constrained Decoding / 14
expertNewOpenAIAnthropicGoogle DeepMind

Choose the speculative draft length per request at runtime. What signal, what policy, and how much does it beat a fixed gamma?

Code wants eight draft tokens and chat wants two, and a fixed gamma serves both badly. The answer simulates three policies on a mixed workload, finds the gain is 5% with a cheap drafter and 19% with an expensive one, then explains why batching takes most of it away.

Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

Code wants eight draft tokens and chat wants two, and a fixed gamma serves both badly. The answer simulates three policies on a mixed workload, finds the gain is 5% with a cheap drafter and 19% with an expensive one, then explains why batching takes most of it away.

10 answers per topic instead of 5, plus saved progress and bookmarks · costs nothingor unlock all 136 remaining answers · ₹2,000 / $25

Every worked answer and derivation in all 12 topics, every concept page, and the full practice-test bank.

6 months · One payment · No auto-renewal

UP NEXT ON YOUR JOURNEY
Next in this trackDesign speculative decoding for a 70B fleet whose batch size swings from 1 at night to 256 at peak. What runs at each load level?Popular right nowWhy is batch-1 decode on a 70B model bandwidth-bound, and what is its latency floor on two H100s?Popular right nowDerive the FLOPs per token for a dense transformer from its config. Where does the 2N rule stop being accurate?
DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.