Advanced

Request Scheduling Policies: FCFS, Shortest-First, Priority and Fair Share

An LLM server's scheduling policy orders the waiting queue: first come first served, shortest prompt first, priority classes, cache-aware ordering or fair share by tokens. The order moves mean and tail TTFT by large factors at the same throughput, decides which requests can starve, and does nothing for the decode speed of requests already running.

Get full Premium access · $25

Every worked answer and derivation in all 12 topics, every concept page, the full practice-test bank, and every code drill’s edge-case tests and reference solution.

6 months · One payment · No auto-renewal

RELATED CONCEPTS

Iteration-Level Scheduling (Continuous Batching)Chunked Prefill and the Token Budget Preemption: Swap Versus Recompute SGLang's Radix Cache and Overlap Scheduler Prefix-Aware and KV-Aware Routing Latency Decomposition: Queue, Prefill, Decode

LESSONS THAT TEACH THIS

Inference for RL rollouts: preserve the training signalBound work before it goes stale15 min→

PRACTICE THIS IN REAL QUESTIONS

Batching, Scheduling & AdmissionQuantify what continuous batching buys over static batching, then pick the chunked-prefill token budget and defend it.→Performance Debugging & BenchmarkingAfter a prompt-template deploy, p99 TTFT went from about 100 ms to over a second while GPU utilization did not move. Debug it.→Inference System DesignDesign the request path for an LLM API: gateway, router, engines, streaming. Where is each SLO measured, and what does each hop cost?→Inference Physics & RooflineWhy does prefill turn compute-bound after a few hundred tokens on an H100, and how long does a 4K-token prompt take on a 70B model?→Batching, Scheduling & AdmissionWhere does head-of-line blocking come from under FCFS, when does shortest-job-first help, and how does it work without job lengths?→Inference System DesignDesign a multi-tenant LLM platform with per-tenant token quotas. How do you stop one tenant from degrading everyone else's latency?→