12Should RoPE run inside the attention kernel or as a separate pass? Count the bytes saved, then find the precision trap at long positions.▼hardNewMetaNVIDIAFireworks AI◆ premiumFusing RoPE saves a read and a write of Q and K, which is real but small in prefill. The precision bug is not small: one tensor in the wrong dtype scrambles positional information by a few hundred tokens. The answer sizes both.Open full answer →
07Implement RoPE for a batched decode step where every row sits at a different position, and verify it against the full-sequence version.▼mediumNewMetaMistral AINVIDIA○ sign inRoPE is a rotation per pair of dimensions. The bugs live in which dimensions pair up, which position each row of a ragged batch is at, what left padding does to position ids, and the precision of the angle at position 100,000.Open full answer →