Skip to content
← Quantization & Numerics / 11
hardNewOpenAIAnthropicGoogle DeepMind

The same prompt gives different logits at batch 1 and batch 64 on an FP8 server. Why, and how can the kernels prevent it?

Reduction order explains the last-bit drift, but a quantized server has a second, much larger leak: a scale computed over the whole batch lets one user's activations change another user's logits. The answer separates the two and fixes both.

Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

Reduction order explains the last-bit drift, but a quantized server has a second, much larger leak: a scale computed over the whole batch lets one user's activations change another user's logits. The answer separates the two and fixes both.

10 answers per topic instead of 5, plus saved progress and bookmarks · costs nothingor unlock all 136 remaining answers · ₹2,000 / $25

Every worked answer and derivation in all 12 topics, every concept page, and the full practice-test bank.

6 months · One payment · No auto-renewal

UP NEXT ON YOUR JOURNEY
Next in this trackFit a 70B model on one 80 GB H100 with room left for KV cache. Choose the weight and KV formats and show the arithmetic.Next in this trackYour serving team does not own pretraining. When is quantization-aware training worth it over post-training quantization?Next in this trackA quantized MoE holds its benchmark scores but loses accuracy on rare domains, traced to rarely routed experts. Explain why and fix it.
DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.