03Pipeline parallelism for inference: why does it raise throughput but never lower per-token latency? Work the bubble arithmetic.▼mediumNewMetaNVIDIATogether AIunlockedA token still has to pass through every layer in order, so splitting the layers across nodes cannot make one token faster. The answer shows what pipelining buys instead, prices the bubble, and finds the all-reduce latency at which it beats cross-node tensor parallelism.Open full answer →
08Serve Llama 3.1 405B: TP=8 on one H100 node, or TP=4 times PP=2 across two nodes? Work the latency and the memory.▼hardNewMetaNVIDIATogether AI○ sign inSame eight GPUs, same weights per GPU, same KV capacity. One layout pays 252 all-reduces per token, the other pays for every layer twice in wall-clock time. The answer works both, then finds the precision choice that forces the second.Open full answer →
11A two-node deployment loads weights, reports healthy, then hangs on the first request. Debug NCCL and the engine.▼hardNewNVIDIAMetaTogether AI◆ premiumA hang gives you no error, no log line and no metric. The answer shows how to make every rank say where it is stuck, why a wrong network transport makes things slow but rarely hangs them, and the order that finds the cause in minutes.Open full answer →