TL;DR: A BM by BN output tile loads (BM + BN) x BK operands per K-step and does 2 x BM x BN x BK FLOPs, so its intensity is BM x BN / (BM + BN): 64 FLOP/B for 128 by 128, 85 for 128 by 256. Bigger tiles cut operand traffic (17.2 GB versus 12.9 GB for an 8,192-cubed BF16 GEMM), and on an H100 they stop at two walls: the FP32 accumulator (128 by 256 is 32,768 registers, half the SM's 65,536) and the 227 KB shared-memory limit per block (four 48 KB pipeline stages). BK = 64 comes from 64 BF16 values being exactly one 128-byte swizzled row.
How to approach it
Treat each tile dimension as its own constraint. M and N set reuse and therefore bandwidth demand; K sets the pipeline stage size and must match the shared-memory layout the tensor cores read; the totals must fit registers and shared memory with enough stages to hide latency. Ask the precision and the problem shape early, because a decode GEMM with M of 16 needs a different answer than prefill.
A strong answer
Why the tile must be large in M and N. Each K-step brings a BM by BK slice of A and a BK by BN slice of B into shared memory and does BM x BN x BK multiply-adds with them:
64 x 64 -> 32.0 FLOP/B
128 x 128 -> 64.0 FLOP/B
128 x 256 -> 85.3 FLOP/B
These bytes come mostly from L2, since neighboring tiles share operands, but the demand is real. For M = N = K = 8,192 in BF16, every A element is reloaded once per tile column and every B element once per tile row:
traffic = 2 B x M x K x (N / BN) + 2 B x K x N x (M / BM)
128 x 128 -> 17.2 GB 128 x 256 -> 12.9 GB
GEMM = 2 x 8,192^3 = 1.10 TFLOP -> 1.11 ms at 989 TFLOP/s dense BF16
needed = 17.2 GB / 1.11 ms = 15.5 TB/s vs 12.9 GB / 1.11 ms = 11.6 TB/s
That is several times the 3.35 TB/s of HBM, which is why tile size, L2 hit rate and tile rasterization (launching tiles in an order that keeps operands in L2) are first-order for GEMM performance.
Why BK = 64. Hopper's WGMMA instruction for BF16 is m64nNk16: one warpgroup (four warps) computes a 64 by N slice, N a multiple of 8 up to 256, with K = 16 per instruction. A BK of 64 is four instructions per stage, and 64 BF16 values are 128 bytes, exactly one row of TMA's 128-byte swizzle mode, which is the layout that lets WGMMA read operands from shared memory without bank conflicts. For FP8, the same 128-byte row holds BK = 128.
What stops the tile growing. Two budgets, per SM:
accumulator, 128 x 256 FP32 = 32,768 values = 128 KB of registers
split over 2 consumer warpgroups (256 threads) = 128 registers per thread
register file = 65,536 x 4 B = 256 KB per SM; max 255 per thread
shared memory stage, 128 x 256 tile, BK = 64: (128 + 256) x 64 x 2 B = 48 KB
227 KB per block / 48 KB = 4 stages (plus room for the epilogue)
128 x 128 stage = 32 KB -> up to 7 stages
The register wall is why Hopper kernels are warp-specialized. CUTLASS's cooperative SM90 kernel runs one producer warpgroup that only issues TMA loads and two consumer warpgroups that issue WGMMA, and it rebalances registers at run time: the producer drops to 40 registers per thread, the consumers rise to 232.
128 threads x 40 + 256 threads x 232 = 5,120 + 59,392 = 64,512 <= 65,536
The stage count matters because each stage covers memory latency. With 4 stages, the TMA for stage k + 3 is in flight while WGMMA works on stage k.
| Tile (BF16, Hopper) | Intensity | Stage | Max stages | Accumulator regs per consumer thread |
|---|---|---|---|---|
| 64 x 64 x 64 | 32 | 16 KB | 14 | 16 |
| 128 x 128 x 64 | 64 | 32 KB | 7 | 64 |
| 128 x 256 x 64 | 85 | 48 KB | 4 | 128 |
| 256 x 256 x 64 | 128 | 64 KB | 3 | 256 (does not fit) |
Why the tile can be too large. Fewer, larger tiles mean fewer work units for 132 SMs. A 640 by 7,168 GEMM makes 140 tiles of 128 by 256, which runs as two waves with the second nearly empty; smaller tiles or a stream-K schedule recover it.
The reversal condition is a skinny M. In decode, M is the batch (often 1 to 64). A 128-row tile wastes most of the tensor-core work and the GEMM is bandwidth-bound anyway, so kernels use BM of 64 or smaller, swap A and B so the large weight dimension maps to the 64-row WGMMA side, or split K to create enough tiles. Tile choice is per shape, which is why libraries ship heuristics and autotuned tables rather than one tile.
What interviewers probe next
"Why not keep the accumulator in shared memory and make the tile bigger?" Then every MMA would read and write the accumulator through shared memory, multiplying its bandwidth demand; Blackwell solves this differently with dedicated tensor memory for the accumulator.
"Now it is FP8." Bytes per element halve, so a 48 KB stage holds BK = 128 and intensity per byte doubles, but the tensor core peak doubles to 1,979 TFLOP/s dense, so the bandwidth demand per second is unchanged.
"How do you pick between 128 by 256 and 256 by 128?" By the shape: put the long dimension of the problem on the long side of the tile so edge tiles waste less, and check the tile count against 132 SMs.
How it's scored
Weak: says tiles are "chosen to fit the cache" or picks sizes by habit, with no reuse arithmetic.
Hire: derives BM x BN / (BM + BN), sizes the shared-memory stages against 227 KB, and connects BK to the instruction shape.
Strong Hire: computes the accumulator's register cost, explains warp specialization and register reallocation as the answer to it, ties BK to the 128-byte swizzle, and names skinny M and wave quantization as reasons to pick a smaller tile.
Common mistakes
- Computing tile intensity against the HBM ridge of 295 FLOP/B and concluding every tile is memory-bound, ignoring L2 reuse between tiles.
- Forgetting the accumulator lives in registers and sizing tiles by shared memory alone.
- Using a 128-row tile for decode-sized M and blaming the tensor cores for low utilization.
- Picking BK for reuse; K does not change intensity, only stage size and pipeline depth.