Skip to content
← Implement-from-Scratch Coding / 29
hardNewNVIDIAAMDMeta

In CUDA, count 1M MoE routing pairs per expert and scatter-add expert outputs back into tokens when one expert is hot.

One atomicAdd per element passes every test and puts 131,072 updates on a single address when the router collapses. The answer counts the serialized updates, privatizes them per warp and per block, and then removes the atomics from the combine entirely with an index the sort already built.

Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

One atomicAdd per element passes every test and puts 131,072 updates on a single address when the router collapses. The answer counts the serialized updates, privatizes them per warp and per block, and then removes the atomics from the combine entirely with an index the sort already built.

10 answers per topic instead of 5, plus saved progress and bookmarks · costs nothingor unlock all 213 remaining answers · $25

Every worked answer and derivation in all 12 topics, every concept page, the full practice-test bank, and every code drill’s edge-case tests and reference solution.

6 months · One payment · No auto-renewal

UP NEXT ON YOUR JOURNEY
Next in this trackImplement a LoRA linear layer by hand, then decide whether to merge the adapter into a BF16 base and into an FP8 base.Next in this trackImplement DeepSeek V3's MoE router from its config, then group 4,096 tokens by expert. What must the tests catch?Next in this trackImplement top-k sparse decode attention over a paged cache and prove it equals dense attention under the same mask.
DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.