Advanced

Serving Engine Knobs and Their Defaults

A handful of engine settings decide how much KV fits, how long a step can get and how many requests run at once. This page lists the defaults in vLLM v0.31.0 and SGLang v0.5.21 as read from source, shows which knob binds first for Llama 3.3 70B on four H100s, and gives the order in which to tune them.

Get full Premium access · ₹2,000 / $25

Every worked answer and derivation in all 12 topics, every concept page, and the full practice-test bank.

6 months · One payment · No auto-renewal

RELATED CONCEPTS

Chunked Prefill and the Token Budget The KV Cache: Bytes per Token and the Memory Budget The vLLM V1 Engine Architecture SGLang's Radix Cache and Overlap Scheduler CUDA Graphs and Launch Overhead in Decode Capacity Planning in Tokens per Second

PRACTICE THIS IN REAL QUESTIONS

KV Cache & GPU MemoryDerive the KV cache footprint for MHA, GQA and MLA, then tell me how many 8K-token conversations fit on four H100s.→Batching, Scheduling & AdmissionQuantify what continuous batching buys over static batching, then pick the chunked-prefill token budget and defend it.→Serving Engine InternalsWalk one request through vLLM V1, from the HTTP call to the streamed token. Which process owns each step, and why is it split that way?→Performance Debugging & BenchmarkingAfter a prompt-template deploy, p99 TTFT went from about 100 ms to over a second while GPU utilization did not move. Debug it.→Speculative & Constrained DecodingWhat do temperature, top-k and top-p each do to the next-token distribution, and in what order does a serving engine apply them?→Serving Engine InternalsWhat do gpu_memory_utilization, max_model_len and max_num_seqs each control in vLLM, and how do they interact at startup?→