← 🟩 GPU Performance
Advanced
Bytes in Flight: Little's Law for GPU Memory
Little's law says sustained memory bandwidth equals the bytes a kernel keeps outstanding divided by memory latency. On an H100 SXM that is about 2 MB across the GPU, or 15 KB per SM, and it decides whether a bandwidth-bound decode kernel can reach peak at all, before any question of coalescing or caching.
Get full Premium access · ₹2,000 / $25
Every worked answer and derivation in all 12 topics, every concept page, and the full practice-test bank.
6 months · One payment · No auto-renewal
RELATED CONCEPTS
PRACTICE THIS IN REAL QUESTIONS
Inference Physics & RooflineWhy is batch-1 decode on a 70B model bandwidth-bound, and what is its latency floor on two H100s?→Attention KernelsDerive online softmax, prove it is exact, and explain why it makes FlashAttention fast without reducing FLOPs.→CUDA, Triton & GPU PerformanceYour decode GEMV kernel reaches only about half of HBM bandwidth on an H100. Find the reasons and get it above 80%.→CUDA, Triton & GPU PerformanceIn a decode GEMV over a row-major weight matrix, which thread-to-element mapping coalesces, and why does the transposed one not?→Batching, Scheduling & AdmissionUse Little's law on an LLM server: how many requests are in flight at 10 per second, and does the KV cache hold them?→Attention KernelsImplement online softmax for a whole tile of queries, not one query, and write the rescaling step in matrix form.→