Skip to content
← Speculative & Constrained Decoding / 10
hardNewOpenAIAnthropicDatabricks

How does grammar-constrained decoding build the allowed-token mask at every step, and what does that cost per step at batch 128?

The model never sees the grammar; the sampler does, as a 16 KB bitmask per request per step. The answer builds that mask from an automaton, prices the naive version at ten milliseconds, and shows how precompilation removes almost all of it.

Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

The model never sees the grammar; the sampler does, as a 16 KB bitmask per request per step. The answer builds that mask from an automaton, prices the naive version at ten milliseconds, and shows how precompilation removes almost all of it.

an account raises the free count per topic
UP NEXT ON YOUR JOURNEY
Next in this trackServe JSON-schema outputs at high throughput: what do you precompile, what do you cache, and what is the tokenizer boundary problem?Next in this trackImplement beam search on a paged KV cache. How do forking and pruning work on block tables, and what does beam width 4 cost in memory?Next in this trackSpeculative decoding with a quantized target or a quantized draft: does losslessness still hold, and against which distribution?
DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.