CUDA, Triton & GPU Performance practice test
The memory hierarchy in numbers, coalescing and bank conflicts, occupancy, Hopper TMA and WGMMA, Blackwell tensor memory, decode GEMV versus prefill GEMM, fused MoE and dequant kernels, CUDA graphs, and reading Nsight like an interviewer. The pool for this topic holds 10 easy, 12 medium and 8 hard items. Each answer comes with its reasoning and a link to the question or concept page it was drawn from.
Sample items with answers
When a warp's addresses fall in one contiguous, aligned 128-byte range, the hardware coalesces them into a single request for four 32-byte sectors, so every byte fetched is used. If instead thread i read element 32i, each thread would touch a different sector and the warp would pull 32 sectors (1,024 bytes) to use 128, an 8x waste of bandwidth. A broadcast happens when all threads read the same address. Bank serialization is a shared-memory effect, not a global-memory one.
There are two unrelated products. Triton the language, from OpenAI, lets you write kernels in Python where each program instance handles a block of data; the compiler handles coalescing, shared-memory staging and tensor-core use within the block. NVIDIA Triton Inference Server, now Dynamo-Triton, is a model server that runs engines behind an API. A kernel team means the language. CUTLASS is the GEMM template library. Conflating the two in an interview is a quick signal that a candidate has not used either.
After the test
Picking an option is easier than deriving the answer at a whiteboard. These pages make you derive it: