Derive the FLOPs per token for a dense transformer from its config. Where does the 2N rule stop being accurate?

TL;DR: Every weight in a matrix multiply costs one multiply and one add per token, so a forward pass is about $2N$ FLOPs per token, 139 GFLOP for Llama 3.3 70B once the embedding lookup is excluded. The rule ignores attention over the context, which adds $4 L n_h d_h T$ FLOPs per token and matches the weight term at about 53K tokens of context for that model.

How to approach it

Walk the config one weight matrix at a time and count multiply-adds, then add the term that has no weights: attention scores and the weighted sum over values. Ask whether the question is about one decode step at a given context, a prefill over a prompt, or training (which is three times the forward). Ask whether embeddings are tied, because the input embedding is a lookup and does no arithmetic.

A strong answer

A token's activation vector of width $d$ times a weight matrix of shape $d \times k$ is $d k$ multiplies and $d k$ adds. Summing over every matrix the token passes through gives the familiar rule:

$$\text{FLOPs}_{\text{weights}} = 2 N_{\text{matmul}}$$

where $N_{\text{matmul}}$ counts only parameters that sit in a matrix multiply. For Llama 3.3 70B (80 layers, hidden 8192, 64 query heads, 8 KV heads of 128, SwiGLU width 28,672, vocab 128,256, untied embeddings):

Q proj      = 8192 x (64 x 128)          =  67,108,864
K and V     = 2 x 8192 x (8 x 128)       =  16,777,216
O proj      = (64 x 128) x 8192          =  67,108,864
MLP (SwiGLU)= 3 x 8192 x 28,672          = 704,643,072
per layer   = 855,638,016  ->  x 80      = 68.45e9
LM head     = 8192 x 128,256             =  1.05e9
N_matmul    = 69.50e9          2 N_matmul = 139.0 GFLOP per token
sanity: config total N = 70.55e9; the 1.05e9 input embedding is a gather, not a GEMM,
        so the naive 2N = 141.1 GFLOP overcounts by 1.5%

The MLP is 82% of the per-layer weights, which is why FFN shape decisions move inference cost more than attention head counts do. Norms, RoPE, SiLU and residual adds are $O(d)$ per layer, a few million operations for the whole 70B model, roughly five orders of magnitude below the GEMMs.

The term the rule leaves out is attention itself. A token at position $T$ computes one score against every cached key and one weighted sum over every cached value, per query head:

$$\text{FLOPs}_{\text{attn}} = 4 \cdot L \cdot n_h \cdot d_h \cdot T$$

Note it uses query heads. GQA shrinks the cache, not this arithmetic: 8 KV heads still serve 64 query heads, each doing its own dot products.

attn per context token = 4 x 80 x 64 x 128          = 2,621,440 FLOPs
at T = 8,192           = 21.5 GFLOP   -> 15% of the 139.0 GFLOP weight term
at T = 32,768          = 85.9 GFLOP   -> 62%
crossover              = 139.0e9 / 2,621,440         = 53,026 tokens
Llama 3.1 8B crossover = 15.01e9 / 524,288           = 28,628 tokens
sanity: smaller models cross over earlier, because weights scale with d^2
        and attention with d

Two points earn the senior signal. First, FLOPs are not time. On two tensor-parallel H100 SXMs at batch 1, each GPU does 69.5 GFLOP, 0.07 ms at the 989 TFLOP/s dense BF16 peak, while reading its 70.6 GB of weights takes 21 ms at 3.35 TB/s, so FLOP counting predicts prefill and large-batch decode, not small-batch latency. Second, the attention term at decode is bandwidth-bound for a separate reason: its FLOPs scale with $T$, but so do the KV bytes it reads, so long context hurts decode through bytes long before it hurts through arithmetic.

The reversal condition: if the workload is short-context chat (prompts and outputs under a few thousand tokens on a 70B), $2N_{\text{matmul}}$ is within a few percent and adding attention is false precision. Once mean context passes roughly a tenth of the crossover, about 5K tokens here, quote the attention term or the estimate is wrong by more than 10%.

What interviewers probe next

"Now it is a 4,096-token prefill. Total FLOPs?" Weights: 4,096 × 139.0 GFLOP = 569 TFLOP. Causal attention averages half the prompt: 2.62M × 4,096² / 2 = 22 TFLOP, so about 591 TFLOP, attention 3.7% of it.

"Does GQA reduce the FLOPs?" Only in the K and V projections: 16.8M parameters per layer instead of MHA's 134M, which saves 117M of an MHA layer's 973M, about 12%. The score and value arithmetic is unchanged because every query head still attends.

"What about the LM head at decode?" It is 1.05B parameters, 1.5% of FLOPs, but its output is a 128,256-wide logit vector per token, and sampling over that becomes visible at large batch.

"Where would 6N be wrong for training?" With activation recomputation the forward runs twice, so the count is closer to 8N, and attention adds its own backward term at long sequence lengths.

How it's scored

Weak: quotes "2 FLOPs per parameter" with no derivation, counts the input embedding, or claims GQA cuts attention FLOPs by 8x.

Hire: builds 139 GFLOP from the per-matrix shapes, adds the $4 L n_h d_h T$ attention term, and computes a crossover context length.

Strong Hire: separates FLOPs from time by showing batch-1 decode spends 300x longer on bytes than arithmetic, applies $T/2$ for causal prefill, and states the context length below which 2N is good enough.

Common mistakes

• Using KV heads instead of query heads in the attention FLOP term, which understates it 8x for Llama 70B.
• Counting a full $T^2$ for prefill attention when FlashAttention-style kernels skip fully masked tiles, doubling the attention estimate.
• Forgetting that the SwiGLU MLP has three matrices, not two, which drops a third of the FFN cost.
• Converting FLOPs to milliseconds at peak TFLOP/s for a decode step, which predicts a latency 300x too optimistic.