Skip to content
← Serving Engine Internals / 08
mediumNewOpenAIAnthropicFireworks AI

Why is streaming detokenization harder than decoding each token, and what breaks with multi-byte characters and stop strings?

Decode each token on its own and an emoji streams as two replacement characters. The answer builds the offset-window decoder that fixes it, runs it, and adds the stop-string holdback that most streaming bugs come from.

Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

Decode each token on its own and an emoji streams as two replacement characters. The answer builds the offset-window decoder that fixes it, runs it, and adds the stop-string holdback that most streaming bugs come from.

an account raises the free count per topic
UP NEXT ON YOUR JOURNEY
Next in this trackHow does an engine batch requests for different LoRA adapters in one forward pass, and what do Punica-style shrink and expand kernels do?Next in this trackWith JSON-schema decoding on, where in the engine step is the grammar mask computed and applied, and what does it cost?Next in this trackYou upgraded the serving engine with the same flags and throughput fell. Which defaults do you diff first, and how do you prove it?
DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.