Skip to content
← Implement-from-Scratch Coding / 10
mediumNewNVIDIAGoogle DeepMindMeta

Implement a ring all-reduce as a simulation over p ranks, verify it against a direct sum, and count the bytes each rank sends.

Two loops and an index formula, and most attempts get the index wrong by one. The answer writes reduce-scatter and all-gather over a simulated ring, proves the result matches a direct sum when p does not divide the length, and turns the byte count into the reason small messages avoid the ring.

Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

Two loops and an index formula, and most attempts get the index wrong by one. The answer writes reduce-scatter and all-gather over a simulated ring, proves the result matches a direct sum when p does not divide the length, and turns the byte count into the reason small messages avoid the ring.

an account raises the free count per topic
UP NEXT ON YOUR JOURNEY
Next in this trackImplement beam search over a paged KV cache, forking beams by sharing blocks and copying only the tail on write.Next in this trackImplement a per-tenant rate limiter for an LLM API that limits tokens, not requests, using a token bucket.Next in this trackImplement a KV block allocator that shares prefix blocks across requests by chained hashes, with LRU eviction of cached free blocks.
DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.