scheduler
Inference interview questions tagged scheduler, across every topic.
5 questions · 2 unlocked for you
Concepts behind "scheduler"
Concept pages for the mechanisms these questions lean on.
Foundational
Iteration-Level Scheduling (Continuous Batching)Iteration-level scheduling, sold as continuous or in-flight batching, re-forms the batch at every forward step instead of once per batch of requests. Finished sequences leave and queued ones join on the next step, which decides how much of a GPU's decode capacity real traffic with uneven output lengths can use.⏱️ Batching & Scheduling
Foundational
The vLLM V1 Engine ArchitecturevLLM's V1 engine splits serving into an API-server process, an EngineCore process that owns the scheduler and the KV cache manager, and one worker per GPU. This page maps each component to its data structures in v0.31.0, works out the KV block pool for an 8B model on an H100, and explains why the split exists: Python overhead has to hide under a 4.8 ms decode step.⚙️ Serving Engines
Advanced
Chunked Prefill and the Token BudgetChunked prefill splits a long prompt into pieces that ride along with decode tokens, under a per-step token budget such as vLLM's max_num_batched_tokens. The budget controls how much prompt work shares a step with decode, trading prompt progress against token gaps without guaranteeing a fixed latency.⏱️ Batching & Scheduling🔒 Premium
Advanced
Preemption: Swap Versus RecomputeWhen the KV pool runs out mid-decode, the scheduler must evict a running sequence: either swap its cache to host memory over PCIe and copy it back later, or free it and recompute it with a fresh prefill. The cost of each depends on model size, KV bytes per token and whether prefix caching still holds the freed blocks.⏱️ Batching & Scheduling🔒 Premium
Advanced
SGLang's Radix Cache and Overlap SchedulerSGLang stores every cached prefix in a radix tree keyed by token ids, so a hit can end at any token rather than at a block boundary, and its overlap scheduler prepares the next batch while the GPU runs the current one. This page walks both through the v0.5.21 source: matching, lock references, leaf-first LRU eviction, the 128-request cutoff on prefix-aware scheduling, and memory defaults.⚙️ Serving Engines🔒 Premium