Skip to content
← Distributed & Disaggregated Inference / 22
hardNewSarvam AITogether AIFireworks AI

Serve Llama 3.3 70B in FP8 on a node of four L40S GPUs with no NVLink: TP=4, two TP=2 replicas, or pipeline parallel?

Every tensor-parallel rule of thumb assumes NVLink. On PCIe-only GPUs the all-reduce is slower and one of vLLM's fast paths switches off, yet the answer for a 70B model is still TP=4. The answer shows why, prices the communication, and finds the model size where two replicas win instead.

Updated Nov 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

Every tensor-parallel rule of thumb assumes NVLink. On PCIe-only GPUs the all-reduce is slower and one of vLLM's fast paths switches off, yet the answer for a 70B model is still TP=4. The answer shows why, prices the communication, and finds the model size where two replicas win instead.

10 answers per topic instead of 5, plus saved progress and bookmarks · costs nothingor unlock all 213 remaining answers · $25

Every worked answer and derivation in all 12 topics, every concept page, the full practice-test bank, and every code drill’s edge-case tests and reference solution.

6 months · One payment · No auto-renewal

UP NEXT ON YOUR JOURNEY
Next in this trackThe disaggregated deployment benchmarks 80% faster than the aggregated one. Prove it is real before anyone rolls it out.Next in this trackRecord MoE routing in the rollout engine so the trainer can replay it. What does it cost per token, and what breaks?Next in this trackTwo SGLang data-parallel ranks deliver well under twice one rank, and half the requests have a bad TTFT. Diagnose it.
DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.