08Why is streaming detokenization harder than decoding each token, and what breaks with multi-byte characters and stop strings?▼mediumNewOpenAIAnthropicFireworks AI○ sign inDecode each token on its own and an emoji streams as two replacement characters. The answer builds the offset-window decoder that fixes it, runs it, and adds the stop-string holdback that most streaming bugs come from.Open full answer →
14Implement a minimal autoregressive generate loop that streams text and honors stop sequences, including ones split across tokens.▼mediumNewOpenAIAnthropicPerplexity◆ premiumThe stop string arrives as three tokens, the euro sign arrives as two, and the client has already been sent half of both. The answer holds back exactly the text that could still become a stop, decodes bytes incrementally, and tests the stream against a reference that sees the whole output at once.Open full answer →