Skip to content
← Distributed & Disaggregated Inference / 14
mediumNewOpenAIAnthropicMeta

You need to get 500 GB of weights onto 1,000 GPUs as fast as possible. What is the lower bound, and what design gets close to it?

Pulling every replica's copy from object storage turns a 500 GB model into 62.5 TB of reads. The answer finds the three bounds that actually limit a fleet load, shows a pipelined broadcast reaching the tightest one, and lists the startup work that hides behind the bytes.

Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

Pulling every replica's copy from object storage turns a 500 GB model into 62.5 TB of reads. The answer finds the three bounds that actually limit a fleet load, shows a pipelined broadcast reaching the tightest one, and lists the startup work that hides behind the bytes.

10 answers per topic instead of 5, plus saved progress and bookmarks · costs nothingor unlock all 136 remaining answers · ₹2,000 / $25

Every worked answer and derivation in all 12 topics, every concept page, and the full practice-test bank.

6 months · One payment · No auto-renewal

UP NEXT ON YOUR JOURNEY
Next in this trackDesign the parallelism plan for a trillion-parameter MoE when your fleet is half H100 nodes and half B200 nodes.Popular right nowWhy is batch-1 decode on a 70B model bandwidth-bound, and what is its latency floor on two H100s?Popular right nowDerive the FLOPs per token for a dense transformer from its config. Where does the 2N rule stop being accurate?
DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.