Skip to content
← Attention Kernels / 13
mediumNewTogether AIFireworks AICohere

Variable-length batches without padding: what is cu_seqlens, how does the kernel index it, and what is the bug that mixes sequences?

Packing removes padding but puts different requests' tokens side by side in one tensor, so the kernel's tile boundaries no longer match sequence boundaries. The answer shows the indexing and a bug that a causal test can never catch.

Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

Packing removes padding but puts different requests' tokens side by side in one tensor, so the kernel's tile boundaries no longer match sequence boundaries. The answer shows the indexing and a bug that a causal test can never catch.

10 answers per topic instead of 5, plus saved progress and bookmarks · costs nothingor unlock all 136 remaining answers · ₹2,000 / $25

Every worked answer and derivation in all 12 topics, every concept page, and the full practice-test bank.

6 months · One payment · No auto-renewal

UP NEXT ON YOUR JOURNEY
Next in this trackYour attention kernel is exact in FP32 and wrong in BF16 at long context, but fine at short context. Find the accumulation bug.Next in this trackDesign attention for a 1M-token context on one 8-GPU node: the kernel, the parallelism across GPUs, and the memory plan.Next in this trackA decode-attention kernel wins at 128K context. What evidence would justify enabling it for other shapes?
DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.