hardNewNVIDIAMetaSnowflake

An INT4 AWQ checkpoint of a 72B model runs slower than BF16 at batch 32 on vLLM. How do you find which kernel is running?

INT4 can shrink weight traffic without improving the full step. Compare the byte model with a kernel trace, distinguish failed startup from a running fallback, and calculate how much materialized dequantization could explain.

Updated Nov 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

10 answers per topic instead of 5, plus saved progress and bookmarks · costs nothingor unlock all 213 remaining answers · $25

Every worked answer and derivation in all 12 topics, every concept page, the full practice-test bank, and every code drill’s edge-case tests and reference solution.

6 months · One payment · No auto-renewal

UP NEXT ON YOUR JOURNEY

Next in this trackNVFP4 scales each block's max to 6. Why can scaling some blocks to 4 lower the error, and how do you choose per block?Next in this trackMoving rollouts and trainer from BF16 to FP16 shrinks their logprob gap about 8x; deterministic mode does nothing. Why?Popular right nowWhy is batch-1 decode on a 70B model bandwidth-bound, and what is its latency floor on two H100s?

EDITOR'S NOTE

Ask for a successful startup record and the actual per-layer trace. A roofline estimate suggests a bottleneck; neither a slow total nor a fallback warning proves which kernel ran.

DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.