14Rank the decode-time arithmetic intensity of the QKV projection, attention, the MLP and the LM head. Which ones can fusion help?▼expertNewNVIDIAOpenAIAnthropic◆ premiumEvery weight GEMM ties, attention sits at a constant the batch cannot move, and the ranking flips at batch 8. The answer then separates what fusion can fix from what only the model architecture or the batch can.Open full answer →
12Should RoPE run inside the attention kernel or as a separate pass? Count the bytes saved, then find the precision trap at long positions.▼hardNewMetaNVIDIAFireworks AI◆ premiumFusing RoPE saves a read and a write of Q and K, which is real but small in prefill. The precision bug is not small: one tensor in the wrong dtype scrambles positional information by a few hundred tokens. The answer sizes both.Open full answer →
07Write a fused RMSNorm kernel in Triton and count its memory traffic against the unfused PyTorch version.▼mediumNewOpenAIMetaTogether AI○ sign inRMSNorm does almost no arithmetic, so its cost is bytes, and the eager PyTorch version moves nine times more of them than it needs to. The answer writes the one-pass kernel, tests it on awkward hidden sizes, and says where fusion stops paying.Open full answer →
08Fused MoE kernels sort tokens by expert before the GEMM. Why, and what does a grouped GEMM do that a loop over experts cannot?▼hardNewNVIDIATogether AIFireworks AI○ sign inA Python loop over 256 experts is correct and slow for three separate reasons, only one of which is launch count. The answer builds the expert-sorted layout the fused kernel runs on and counts what it costs in padding.Open full answer →