TL;DR: For DeepSeek V3 each token sends 8 copies of its 7,168-wide hidden state out (FP8, 57,344 B) and gets 8 back (BF16, 114,688 B) in each of 58 MoE layers: 9.98 MB per token, 2.55 GB per step at batch 256. On EP=32 across four H100 nodes, three quarters of that crosses the NICs, about 1.2 ms per step at 50 GB/s per GPU, and dispatching once per destination node cuts the network share by more than half.
How to approach it
Write the per-copy bytes first (hidden size times bytes per element, for dispatch and combine separately), then multiply by top-k, MoE layers and tokens. Only after that split the traffic by link: what stays on the GPU, what rides NVLink inside the node, and what crosses the network. Ask the EP degree, the dispatch precision, whether the batch of 256 is per GPU or per instance, and whether the dispatch kernel deduplicates by node. Each changes the answer by a factor of two or more.
A strong answer
Bytes per token. DeepSeek V3 has 61 layers; the first 3 are dense, so 58 carry the MoE. The shared expert runs locally on every GPU and sends nothing. Each routed expert copy needs the token's hidden state on the way out and returns a weighted output on the way back. Dispatch in FP8 and combine in BF16 is the usual choice, because the expert GEMM consumes FP8 but the weighted sum wants precision:
dispatch per copy = 7,168 x 1 B = 7,168 B x 8 experts = 57,344 B
combine per copy = 7,168 x 2 B = 14,336 B x 8 experts = 114,688 B
per token per MoE layer = 172,032 B
per token, 58 MoE layers = 58 x 172,032 B = 9.98 MB
per step, 256 tokens = 256 x 9.98 MB = 2.55 GB
sanity: FP8 activation scales at 1 x 128 tiles add 56 x 4 B = 224 B per copy, 3%
Where the bytes go. Take EP=32 across four 8-GPU H100 nodes, 8 routed experts per GPU, and data-parallel attention so each GPU owns 8 of the 256 tokens. Assume one 400 Gb/s NIC per GPU, 50 GB/s per direction. With uniform routing, a copy stays on its own GPU with probability 1/32, stays in the node with 7/32, and leaves the node with 24/32:
per GPU per step = 2.55 GB / 32 = 79.8 MB
over the NIC = 79.8 MB x 24/32 = 59.9 MB -> / 50 GB/s = 1.20 ms
over NVLink = 79.8 MB x 7/32 = 17.5 MB -> / 450 GB/s = 39 us
per MoE layer = 59.9 MB / 58 = 1.03 MB -> 20.6 us on the NIC
compare: expert weights per GPU = 58 x 8 x 44.0M params x 1 B = 20.4 GB
read at 3.35 TB/s = 6.10 ms
The network adds about a fifth to the expert weight read if it is not overlapped. NVLink traffic is negligible. And 20 µs of payload per layer is small enough that the fixed latency of each all-to-all (setup, synchronization, small RDMA writes to 24 peers) can rival it, which is an assumption to measure, not a constant to quote.
Node-aware dispatch. A token whose 8 experts sit on 3 GPUs in the same remote node should cross the network once and fan out over NVLink. That is how DeepSeek's DeepEP kernels and similar implementations are built. Of the 8 copies, 6 go to remote nodes on average, but they land on only 2.71 distinct remote nodes:
P(a given remote node gets none of the 8) = C(192, 8) / C(256, 8)
expected remote nodes hit = 3 x (1 - C(192, 8) / C(256, 8)) = 2.71
network copies: 6.0 -> 2.71, so NIC traffic x 0.45 -> 1.20 ms becomes 0.54 ms
Combine benefits the same way if partial outputs are summed inside the remote node before they come back. Restricting routing to at most nodes per token, which DeepSeek V3 does in training, caps the copies further.
from math import comb
def a2a_bytes(tokens, hidden=7168, top_k=8, moe_layers=58,
dispatch_b=1, combine_b=2):
"""Bytes moved by dispatch plus combine for one decode step, no dedup."""
return tokens * top_k * hidden * (dispatch_b + combine_b) * moe_layers
def remote_nodes_per_token(experts=256, nodes=4, top_k=8):
"""Expected distinct remote nodes hit, uniform routing, experts packed by node."""
per_node = experts // nodes
p_miss = comb(experts - per_node, top_k) / comb(experts, top_k)
return (nodes - 1) * (1 - p_miss)
print(a2a_bytes(256) / 1e9) # 2.554 GB per step
print(remote_nodes_per_token()) # 2.711, matches a 100k-token simulation
assert a2a_bytes(0) == 0
When does the answer flip? If the batch of 256 is per GPU, not per instance, everything scales by 32: 38 ms of NIC time per step without dedup, far above the 6 ms weight read, and the all-to-all becomes the step. Then you shrink EP to fit one NVLink domain if memory allows, deduplicate, and overlap dispatch with compute using two micro-batches. Routing skew also breaks the uniform assumption: a hot expert's GPU receives more than its share, and every rank waits for it.
What interviewers probe next
"Do the same on a single 8-GPU node." All copies ride NVLink: 2.55 GB / 8 × 7/8 = 279 MB per GPU at 450 GB/s is 0.62 ms, but DeepSeek V3 in FP8 needs about 671 GB of weights, which an 8×H100 node cannot hold.
"Why FP8 dispatch but BF16 combine?" The expert GEMM consumes FP8 activations anyway, so quantizing before the send halves the bytes for free. The combine is a weighted sum the residual stream depends on, and its error accumulates over 58 layers.
"What about the all-to-all's latency at batch 1?" Eight copies of 7 KB per layer is a latency-bound message. Per-layer time is then dominated by the fixed cost, and low-latency kernels that write directly into peers' buffers win over bandwidth-optimal ones.
How it's scored
Weak: sizes the traffic as an all-reduce of the hidden state, or forgets the combine direction.
Hire: gets 172 KB per token per layer from top-8, hidden size and the two precisions, multiplies by 58 MoE layers, and splits local, NVLink and network shares.
Strong Hire: compares the network time with the expert weight read, computes the node-dedup saving, and notes that "batch 256" per GPU versus per instance changes the answer by 32x.
Common mistakes
- Multiplying by 61 layers, which includes the three dense layers that send nothing.
- Sending the shared expert's input over the network; it is replicated and local.
- Using one copy per token instead of top-k copies, an 8x undercount.
- Assuming NIC bandwidth applies to every byte, when 8 of 32 destinations are on the same GPU or in the same node.