Build intuition.
Then put it to work.
Follow a sequence through the published library, or rehearse the 25-problem interview review. Work each problem before opening its answer, then use the supporting concepts and experiments to check your reasoning.
Check your starting point25 inference interview problems
Work from the first token to fleet economics. For each problem, state your assumptions, work the numbers, and explain what would change your decision.
- Bring with you
- Transformer fundamentals and working Python. Use the supporting links when a mechanism is unfamiliar.
- Work toward
- Produce a defensible answer to 25 serving problems, with a calculation, a failure case or a test for each. This is a review route, not a promise of mastery in one day.
- Request and memory01Concept · Free
Latency Decomposition: Queue, Prefill, Decode
Trace arrival, tokenization, admission, prefill, sampling and first content. State which clock each latency uses.
- 02Worked question · Premium
A streaming client stops reading. Keep its output bounded without blocking the inference batch.
Implement a bounded SSE outbox, then explain disconnect, timeout and in-flight cancellation.
- 03Worked question · Available
Derive the KV cache footprint for MHA, GQA and MLA, then tell me how many 8K-token conversations fit on four H100s.
Size 100 simultaneous 32K contexts; distinguish KV heads, precision, page rounding and weight memory.
- 04Concept · Premium
Paged KV Memory and Block Tables
Draw the logical-to-physical block table and explain fragmentation, shared prefixes and copy-on-write.
- 05Worked question · Available
Count the HBM bytes FlashAttention reads and writes for one head at N = 8K and d = 128, and compare it with naive attention.
Count the HBM traffic that tiled attention removes, while preserving the attention operation.
- Scheduling and decoding06Worked question · Available
Quantify what continuous batching buys over static batching, then pick the chunked-prefill token budget and defend it.
Compare static and iteration-level batches using a request mix, then impose a decode deadline.
- 07Worked question · Available
Where does head-of-line blocking come from under FCFS, when does shortest-job-first help, and how does it work without job lengths?
Admit a 100K prompt beside short chats. Protect both waiting requests and active decoders.
- 08Worked question · Available
Design prefix-aware routing that keeps cache hits high without overloading the replica that owns a hot prefix. What is your fallback rule?
Choose a replica using reusable prefixes and queued work; say when affinity must yield to load.
- 09Worked question · Available
TTFT is fine but inter-token latency is bad. Decompose the decode step and rank the causes.
Separate weight reads, KV reads, host gaps and mixed prefill before trading throughput for lower ITL.
- 10Worked question · Premium
Design speculative decoding for a 70B fleet whose batch size swings from 1 at night to 256 at peak. What runs at each load level?
Decide when accepted draft work repays verification cost and when drafting should stop.
- Numerics and GPU work11Worked question · Available
AWQ INT4 beat FP8 at batch 1, lost to it at batch 256, and cost 4 points on a coding eval. Explain both and pick a format.
Compare FP8, INT8, INT4 and NVFP4 by storage, kernel support and workload regime; declare a quality gate.
- 12Worked question · Premium
Plan capacity for a 70B fleet that mixes H100 and B200 nodes under one latency SLO. How do you count capacity and route across them?
Shortlist H100, H200, B200 and L40S using whole-replica fit, latency and qualified-request cost.
- 13Worked question · Free account
Here is an Nsight Systems timeline of one decode step. Find the launch gaps, synchronizations and CPU stalls, and price each one.
Use the system timeline to locate idle time, then choose kernel counters for the remaining bottleneck.
- 14Worked question · Premium
An 8B model on one H100 stops scaling at high request rates and the GPU is not saturated. Prove the CPU is the bottleneck, then fix it.
Treat 30% GPU utilization as a symptom. Check offered load, CPU work and synchronization before buying capacity.
- 15Worked question · Free account
Write a fused RMSNorm kernel in Triton and count its memory traffic against the unfused PyTorch version.
Write a masked RMSNorm reduction, verify reference numerics, and benchmark eager and compiled baselines.
- Evidence and operations16Worked question · Premium
Build the dashboard you want open before your next inference incident, and say what decision each panel drives.
Attach an operational decision to latency, queue, KV and prefix metrics. Keep cost denominators explicit.
- 17Worked question · Free account
p99 TPOT spikes every few seconds in a regular pattern while p50 stays flat. Find the cause.
Identify which latency has a 10:1 tail-to-median ratio, then correlate slow requests with step traces.
- 18Worked question · Free account
TensorRT-LLM in 2026: what is the PyTorch backend, how does its in-flight batching admit requests, and when do you still pick it?
Compare pinned vLLM, TensorRT-LLM and SGLang builds on an arrival trace with equal tuning budgets.
- 19Worked question · Premium
Design the rollout of a new model version so that neither quality nor latency regresses in production. How do you know when to promote?
Set quality and performance rejection criteria before a shadow, canary and rollback sequence.
- 20Worked question · Available
Design autoscaling for an LLM fleet whose model takes four minutes to load. What do you scale on, and how much headroom do you hold?
Turn demand growth and cold-start lead time into ready capacity, with a defined overload policy.
- Fleet decisions21Worked question · Available
Design a multi-tenant LLM platform with per-tenant token quotas. How do you stop one tenant from degrading everyone else's latency?
Budget tenant work, protect interactive deadlines and prevent background jobs from starving.
- 22Worked question · Free account
You split prefill and decode onto separate nodes. Compute the KV transfer per request and the network bandwidth it needs.
Derive KV bytes and transfer time, then identify ownership and dependencies on the handoff path.
- 23Worked question · Premium
Design the parallelism plan for a trillion-parameter MoE when your fleet is half H100 nodes and half B200 nodes.
Choose TP, PP and EP from weight residency, active expert work and the actual interconnect.
- 24Worked question · Premium
Cut inference cost per million output tokens by 70% while preserving quality and latency. Can you defend the target?
Test a 70% cost target against a residual-cost model, latency qualification and a quality margin.
- 25Worked question · Available
Size the fleet for a 70B chat API at 200 req/s, 2K tokens in and 300 out, p99 TTFT under 1 s, TPOT under 50 ms. What is the token cost?
Convert one million daily users into peak token demand under explicit activity assumptions; round whole replicas before pricing.
These are curated study paths through existing material, with access shown at each step. For dedicated lessons, exercises and progress tracking, explore the published courses.
Explore the courses