Skip to content
← Attention Kernels / 20
expertNewNVIDIATogether AIBaseten

Sparse MLA decode on a B200 reads FP8 or FP4 KV but multiplies in BF16. Is it bandwidth-bound, and why dequantize?

A low-precision KV cache sounds like a bandwidth story. Count FLOPs per byte for 64 heads sharing one 512-dim entry and the kernel lands on the B200's ridge, where the query and output move more bytes than the cache.

Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

A low-precision KV cache sounds like a bandwidth story. Count FLOPs per byte for 64 heads sharing one 512-dim entry and the kernel lands on the B200's ridge, where the query and output move more bytes than the cache.

10 answers per topic instead of 5, plus saved progress and bookmarks · costs nothingor unlock all 213 remaining answers · $25

Every worked answer and derivation in all 12 topics, every concept page, the full practice-test bank, and every code drill’s edge-case tests and reference solution.

6 months · One payment · No auto-renewal

UP NEXT ON YOUR JOURNEY
Next in this trackWhich attention kernel will vLLM and SGLang actually run on an H100 and a B200, and what silently changes the answer?Next in this trackResearch hands you document masking plus sink tokens to serve. FlexAttention, or fork a FlashAttention kernel?Popular right nowWhy is batch-1 decode on a 70B model bandwidth-bound, and what is its latency floor on two H100s?
DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.