TL;DR: Coalescing is about what the 32 lanes of a warp touch in one load instruction: consecutive lanes must read consecutive addresses. For a row-major
[N, K]weight (PyTorch'snn.Linearlayout), lanes must walk along K inside one row and reduce across the warp at the end; one warp-wide 16-byte load then covers 512 contiguous bytes, 16 full 32-byte sectors. Giving each thread its own row puts the 32 lanes 8,192 bytes apart (K = 4,096 in BF16), so one instruction asks for 32 sectors to use 64 bytes, 6.25% of what it fetches.
How to approach it
Start from the layout, not the math: which dimension is contiguous in memory, and what does one warp instruction touch? Ask for the shape (decode GEMVs are [out, in] with in contiguous), the element type, and the load width. Then decide which dimension the lanes index, and where the reduction happens, because those two choices are linked.
A strong answer
The memory system serves a warp's load in 32-byte sectors, grouped into 128-byte cache lines. A load instruction is coalesced when the addresses of all 32 lanes fall into as few sectors as the bytes requested allow. The cost model is sectors fetched per useful byte.
Take Llama 3.1 8B's q_proj, 4,096 by 4,096 in BF16, stored row-major so that row n is 4,096 contiguous elements:
row length = 4,096 elements x 2 B = 8,192 B
matrix = 4,096 rows x 8,192 B = 33.6 MB
one sector = 32 B = 16 BF16 elements
Mapping A, lanes along K (coalesces). One warp owns one row (or a few). Lane j loads 8 BF16 values (16 bytes) at column k0 + 8j, so the warp reads one contiguous 512-byte run of the row:
bytes per warp load = 32 lanes x 16 B = 512 B
sectors touched = 512 / 32 = 16, every byte used
cache lines = 512 / 128 = 4
Each lane keeps a running FP32 partial sum over its columns as the warp strides along the row, and at the end five __shfl_down_sync steps (16, 8, 4, 2, 1) combine the 32 partials into the output element. The reduction is the price of coalescing, and it costs a handful of instructions per row against 8,192 bytes streamed.
Mapping B, one thread per output row (does not coalesce). Thread n computes y[n] alone and loops over k. At each step the 32 lanes read W[n, k] for 32 consecutive n, which are 8,192 bytes apart:
useful bytes per load = 32 lanes x 2 B = 64 B
sectors requested = 32 (one per lane) = 1,024 B
efficiency = 64 / 1,024 = 6.25%
The sector each lane fetched holds the next 15 elements of its row, so if those sectors stay in L1 until the thread comes back for them, DRAM traffic does not grow 16x. The load/store unit still processes 32 sectors per instruction instead of 4 for an equivalent coalesced 4-byte load, and with many warps per SM the sectors are evicted before reuse. In the profiler this shows as about 32 sectors per request in Nsight Compute's memory tables, against the ideal of 4 for 4-byte loads.
A short counter makes the difference concrete:
import numpy as np
def sectors_per_load(byte_addrs): # distinct 32 B sectors touched
return len(np.unique(np.asarray(byte_addrs) // 32))
K, ELT = 4096, 2 # row-major [N, K], BF16
lanes = np.arange(32)
along_k = 0 * K * ELT + lanes * 16 # row 0, 16 B per lane
per_row = lanes * K * ELT + 0 # rows 0..31, column 0
print(sectors_per_load(along_k), sectors_per_load(per_row)) # 16 32
The transposed mapping is not always wrong. If the weight were stored [K, N] (contiguous along the output dimension), one thread per output column coalesces naturally: at step k the 32 lanes read W[k, n0..n0+31], 64 contiguous bytes, and no cross-lane reduction is needed. The rule that survives every case: give consecutive lanes the contiguous dimension. Which dimension that is depends on the storage layout, and that is why prepacked inference weights (Marlin, TensorRT-LLM) are reordered offline to match the kernel's access pattern.
The reversal condition is a tiny K. When the reduction dimension is shorter than a warp's load (K of a few dozen), a warp-per-row mapping leaves lanes idle, and the better kernel assigns several rows per warp or switches to the [K, N] layout.
What interviewers probe next
"Now the weights are INT4, two per byte." The rule is unchanged but the unit is bytes: a 16-byte lane load now carries 32 weights, so a warp covers 1,024 weights of a row per instruction and the unpacking happens in registers after the coalesced load.
"Batch is 8, not 1." Load the weight tile once and reuse it against all 8 activation vectors from registers or shared memory; the coalescing analysis applies to W, and the activations are small enough to broadcast.
"Your kernel is coalesced but still at 60% of bandwidth." Coalescing fixes waste per byte; reaching peak also needs enough bytes in flight per SM and enough thread blocks to cover all 132 SMs, which is a separate check.
How it's scored
Weak: says "coalescing means threads access memory together" without stating which addresses consecutive lanes touch, or picks mapping B because each thread owns a whole dot product.
Hire: computes the 8,192-byte lane stride and the sectors per instruction for both mappings, and adds the warp-shuffle reduction that mapping A needs.
Strong Hire: states the layout-dependent rule, explains why L1 sometimes hides mapping B's DRAM cost but not its transaction count, names the profiler metric, and knows that weight prepacking exists to make the kernel's access order contiguous.
Common mistakes
- Reasoning about one thread's access sequence (each thread does read its row contiguously) instead of what the 32 lanes touch in a single instruction.
- Assuming PyTorch
nn.Linearweights are[in, out]. They are[out, in], contiguous along the input dimension. - Using 2-byte scalar loads in mapping A: coalesced, but it needs eight times the instructions of 16-byte vector loads to move the same bytes.
- Forgetting the cross-lane reduction, so each lane writes its partial sum as the final output.