The mask is a 16 KB bitmap per request. Whether it costs nothing or stalls every step depends on which thread computes it and what it is allowed to overlap with.
an account raises the free count per topic
← Serving Engine Internals / 10
mediumNewOpenAIAnthropicFireworks AI
With JSON-schema decoding on, where in the engine step is the grammar mask computed and applied, and what does it cost?
The mask is a 16 KB bitmap per request. Whether it costs nothing or stalls every step depends on which thread computes it and what it is allowed to overlap with.
Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.
UP NEXT ON YOUR JOURNEY
Next in this trackYou upgraded the serving engine with the same flags and throughput fell. Which defaults do you diff first, and how do you prove it?Next in this trackllama.cpp versus vLLM: what is each engine built for, and when does the CPU or Apple Silicon path actually win?Next in this trackWrite the main loop of a minimal LLM serving engine: request queue, scheduler, model step, sampling and streaming.
DISCUSSION · 0
Nobody has commented on this answer. Add a correction or another way to work it.