Build intuition.
Then put it to work.
Follow a sequence through the published library, or rehearse the 25-problem interview review. Work each problem before opening its answer, then use the supporting concepts and experiments to check your reasoning.
Check your starting pointGPU & kernel engineering
Connect the memory hierarchy to attention, numerical formats, and a defensible kernel measurement.
8 steps · First concept free · Access shown at each step
- Bring with you
- Tensor shapes, linear algebra, and familiarity with GPU execution.
- Work toward
- Explain a kernel's memory traffic and choose a measurement that matches its use.
- A good fit if
- You write or review GPU code and want to explain a kernel's speed from bytes and FLOPs rather than from a profiler's verdict.
- Not in this path
- Scheduling, routing and fleet design, which the serving path covers.
Check your understanding 4 tasks to try without the answer
Write the online-softmax recurrence and the log-sum-exp merge from memory, then check them against a dense softmax.
Redo if Your merge rescales the output and forgets the running sum, or the reverse.
For a decode kernel, compute the bandwidth floor on a named GPU and say how far a measured time sits from it.
Redo if You judged a bandwidth-bound kernel by its FLOP rate.
Give the range and step size of BF16 and FP8 E4M3 and choose a KV format with a reason.
Redo if You swapped what the exponent and mantissa bits buy.
Specify a kernel benchmark's shapes, clocks, warmup and reported statistic before running it.
Redo if Your result is one timing from one run.
- 01Concept · Free
The GPU Memory Hierarchy in Numbers
Locate data in the memory hierarchy.
- 02Concept · Free
The Online Softmax Recurrence
Derive the state needed for tiled softmax.
- 03Concept · Free
Number Formats for Inference: BF16, FP8, INT4, FP4
Distinguish range, precision, and scaling.
- 04Experiment · Free
Decode economics
Explore how batch size changes a decode bound.
- 05Worked question · Premium
You are about to report one number for your new kernel in a design review. How do you measure it so the number survives scrutiny?
Define the workload before timing a kernel.
- 06Concept · Free
Attention With a KV Cache, From Scratch
Read and reason through cached attention.
- 07Concept · Premium
Split-KV Decoding (Flash-Decoding)
Explain why splitting helps some decode shapes and costs others.
- 08Worked question · Premium
A decode-attention kernel wins at 128K context. What evidence would justify enabling it for other shapes?
Test ragged attention and decide where a candidate kernel earns its dispatch cost.
- Read the question and stop. Write down what you would ask the interviewer, because the missing input is often the point.
- Write the chain on paper: inputs with their units, the substitution, the result and one line that checks it is plausible.
- Open the answer and find the first place your reasoning diverges. That place matters more than whether the final number matched.
- Grade yourself against the Weak, Hire and Strong Hire lines under “How it’s scored”, then fix the step that cost you the grade.
- Change one input, such as the GPU, the precision or the context length, and predict the new answer before you recompute it.
These are curated study paths through existing material, with access shown at each step. For dedicated lessons, exercises and progress tracking, explore the published courses.
Explore the courses