token budget

The gain is not a constant. It is a statistic of your output-length distribution, and the right token budget is a point on the roofline. The answer derives both, then says where the paper number and the real knee disagree.

The default of 1,024 sequences is never reached by a 2K-context workload on an 8B model, and is past the compute ridge for a 256-token one. The answer finds the three ceilings and picks the lowest.

Each draft token is a real token in the step's budget, and verification turns a bandwidth-bound step compute-bound sooner. The answer shows the accounting, the rollback that makes rejection cheap, and the batch size where speculation starts losing.

A scheduler is a loop over three queues and a budget. The answer gets the order of those loops right, shows what the budget costs in milliseconds of TPOT, and tests the invariants a grader will check.