Skip to content
← CUDA, Triton & GPU Performance / 07
mediumNewOpenAIMetaTogether AI

Write a fused RMSNorm kernel in Triton and count its memory traffic against the unfused PyTorch version.

RMSNorm does almost no arithmetic, so its cost is bytes, and the eager PyTorch version moves nine times more of them than it needs to. The answer writes the one-pass kernel, tests it on awkward hidden sizes, and says where fusion stops paying.

Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

RMSNorm does almost no arithmetic, so its cost is bytes, and the eager PyTorch version moves nine times more of them than it needs to. The answer writes the one-pass kernel, tests it on awkward hidden sizes, and says where fusion stops paying.

an account raises the free count per topic
UP NEXT ON YOUR JOURNEY
Next in this trackFused MoE kernels sort tokens by expert before the GEMM. Why, and what does a grouped GEMM do that a loop over experts cannot?Next in this trackIn a W4A16 GEMM, where does the INT4 unpacking happen, and how does the kernel keep that work hidden under the memory loads?Next in this trackHere is an Nsight Systems timeline of one decode step. Find the launch gaps, synchronizations and CPU stalls, and price each one.
DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.