Good questions start
with deeper understanding.
Keep the mechanism close. Explore the concepts, experiments, and guides behind the serving path.
Prefer a sequence? Find your pathDecode economics
Change the batch, context, and hardware. Compare a sequential phase bound with the memory budget.
KV memory budget
See what remains for the KV cache after stored weights and runtime reservations.
Speculative decoding
Find when useful draft tokens pay for drafting and verification.
Prefill deadlines
Choose a prefill chunk under a decode-token deadline and inspect every iteration.
Prefix ownership
Count physical KV pages when requests share a prefix within separate trust groups.
SLO goodput
Apply latency and success requirements to each request in a synthetic trace.
Context growth
Keep the prefill chunk fixed and watch growing attention work consume the token-gap budget.
Batch scheduling
Compare static and continuous batching on a finite, synthetic request queue.
The inference field guide
Eight pages of worked serving decisions: KV reservations, prefill deadlines, cancellation, speculation, benchmark qualification and kernel investigation.
Serving under pressure: from admission to release
Build a defensible serving policy for one specified worker: reserve KV pages, protect token deadlines, reclaim cancelled work safely, and decide whether a change deserves release.
Reading inference engines: prove what the code preserves
Follow a token through an inference engine, test the state that survives each step, and audit the evidence behind a performance claim. Six lessons turn source reading into a reproducible engineering review.
How to read an inference engine: a route through code, kernels and measurement
A practical reading route through Mini-SGLang, LLMs-from-scratch, Uzu, AIPerf and primary inference courses. Each stop has a specific file or section, an engineering question and an artifact to produce.