TL;DR: MLA stores one 576-wide latent per token per layer, shared by all 128 heads, so tensor parallelism over heads makes every rank hold and read the full cache: TP=8 keeps 8 copies. Switching attention to data parallelism (each GPU owns its own requests) while the MoE runs expert-parallel raises an 8×H200 node's DeepSeek V3 KV capacity from 652k to 4.08M tokens, 6.3x, for about 10 GB of replicated attention weights per GPU.
How to approach it
Start from what each TP rank needs to compute its heads. For GQA it needs its own KV heads, so the cache divides by TP. For MLA every head reads the same latent, so it cannot. Then ask what parallelism the rest of the layer wants: MoE weights are 98% of DeepSeek V3's parameters and are best sharded by expert. Ask the hardware, the context lengths and the batch per GPU, because those decide whether the extra attention weight read costs more than the cache it frees.
A strong answer
The problem. DeepSeek V3's KV cache per token is bytes:
MLA KV per token (BF16) = 61 layers x 576 x 2 B = 70,272 B
GQA comparison: Llama 3.3 70B at TP=8 = 327,680 B / 8 = 40,960 B per GPU per token
MLA at TP=8 = 70,272 B per GPU per token (no division)
In decode, engines run MLA in its absorbed form, where each head's query is projected into the latent space and attends over the shared latent directly. Rank owns 16 of the 128 heads, but all 16 read all 576 dimensions of every cached token. Splitting the latent across ranks would just turn attention into a sharded dot product with an extra reduction. So under TP=8 the cache is replicated 8 times, both in memory and in the bytes read per decode step.
The fix. Give each GPU its own requests, its own full copy of the attention weights and its own cache. Attention then needs no communication. At the MoE, every GPU dispatches its tokens to whichever GPUs own their experts (EP=8 here, 32 routed experts per GPU) and gathers the results, then continues with its own requests.
The capacity arithmetic, on 8×H200 (141 GB each, 92% usable, FP8 weights, BF16 KV):
usable per GPU = 141 GB x 0.92 = 129.7 GB
attention weights, all layers = 61 x 187.1M params = 11.4 GB in FP8
TP=8 everywhere: weights/GPU = 671.0 GB / 8 = 83.9 GB -> 45.8 GB for KV
capacity = 45.8 GB / 70,272 B = 652k tokens, the same on every GPU
DP attention + EP=8: weights/GPU = (671.0 - 11.4) / 8 + 11.4 = 93.9 GB -> 35.9 GB KV
capacity = 8 x 35.9 GB / 70,272 B = 4.08M tokens
ratio = 4.08M / 652k = 6.3x; at 4,096-token contexts: 159 sequences vs 996
It also fixes bandwidth. At 159 sequences of 4,096 tokens, TP makes each GPU read 45.8 GB of cache per step (9.5 ms at 4.8 TB/s); DP attention reads 5.7 GB per GPU (1.2 ms) for the same work.
What it costs. Each GPU now reads all 11.4 GB of attention weights per step instead of 1.4 GB: 2.4 ms against 0.3 ms at 4.8 TB/s. At batch 1 that is a pure loss, because the cache is tiny and the extra weight read is all you pay. Second, the DP ranks are lockstepped by the MoE all-to-all: a rank with no tokens must still join with an empty batch, and a rank holding long-context requests runs slower attention while seven others wait. Third, the scheduler must place requests across ranks by KV load, which is a load balancer that TP never needed.
The decision: DP attention plus EP once the cache, not the weights, limits the batch, which for MLA happens early because TP gives it no relief. Keep TP attention for latency-first, small-batch serving where the 2 ms of extra weight read per step outweighs the freed memory. A GQA model with enough KV heads to divide by TP has much less reason to switch.
Engine specifics, checked October 2026. SGLang v0.5.21 exposes it as enable_dp_attention (off by default), with help text saying the DP size should equal the TP size. vLLM v0.31.0 builds the same layout from data_parallel_size plus enable_expert_parallel (off by default), with the all-to-all backend chosen by all2all_backend, whose default is allgather_reducescatter.
What interviewers probe next
"Why not shard the latent dimension across ranks instead?" Each rank would compute a partial score over 72 of 576 dimensions, and every score would need an all-reduce before the softmax: per layer, per step, on the latency-critical path.
"The node has 8 GPUs but you have 6 long-context users." Two ranks sit idle in attention and still join every all-to-all. You need either a mixed TP-within-DP layout (DP=4 with TP=2 attention) or routing that spreads long requests.
"Does the dense first-three-layers part change anything?" Those MLPs are replicated or tensor-split separately; at 3 x 3 x 7,168 x 18,432 = 1.19B parameters they are 0.2% of the weights, so either choice is cheap.
How it's scored
Weak: assumes TP divides the MLA cache by 8 like a GQA cache, and so sees no reason for DP attention.
Hire: explains that the latent is shared by every head, so TP replicates it, and describes DP attention feeding an EP all-to-all.
Strong Hire: quantifies the capacity gain against the replicated attention weights, names the lockstep and load-balancing costs, and says when TP attention is still better.
Common mistakes
- Counting the KV latent at 512 and forgetting the 64-wide decoupled RoPE key.
- Treating DP attention as free: the attention weights are read once per GPU per step, 8 times the TP total.
- Ignoring empty DP ranks, which still have to participate in every collective.
- Mixing GB and GiB when converting the free memory into tokens, a 7% error at this scale.