Practice tests · 30 items

Serving Engine Internals practice test

vLLM V1, SGLang, TensorRT-LLM and llama.cpp from the inside: the engine core loop, schedulers, KV managers, model runners, CUDA graph capture, detokenisation and streaming, and the configuration knobs whose defaults you must be able to defend. The pool for this topic holds 10 easy, 12 medium and 8 hard items. Each answer comes with its reasoning and a link to the question or concept page it was drawn from.

Configure a test

Topic

Difficulty

Questions

10 in this pool · about 7 min

Reveal answers

Sample items with answers

easy · sample

vLLM v0.31.0 defaults gpu_memory_utilization to 0.92. What does that number limit?

The fraction of free memory after loading weights that becomes the KV cache pool

The KV cache alone; weights and activations are allocated outside the 0.92

✓vLLM's share of each GPU's total memory: weights, activations, graphs and KV

The peak utilization the scheduler targets before it starts to preempt requests

The fraction applies to the GPU's total memory, and everything vLLM allocates comes out of it: weights, the profiled activation peak, CUDA graph memory, then the KV pool, which gets whatever is left. On an 80 GB H100 that is 73.6 GB; Llama 3.1 8B's 16.06 GB of BF16 weights leave about 57.5 GB before activations. Reading it as a share of post-weight free memory overestimates the pool. It is a static allocation, not a scheduling target; the default was 0.9 before v0.20.

easy · sample

In vLLM's V1 engine, which process tokenizes the prompt and detokenizes the output?

The EngineCore process, right before the scheduler admits each request

✓The API-server process, so text never enters the EngineCore process

Each GPU worker, which tokenizes the requests in its own batch shard

The client SDK, since the server accepts and returns only token ids

V1 splits serving into API-server processes, one EngineCore per data-parallel rank and one worker per GPU. The API server applies the chat template, tokenizes, and sends EngineCore a request carrying token ids and sampling parameters; on the way back it detokenizes and streams. EngineCore owns the scheduler and the KV block pool and never touches text. The split keeps Python work off the critical path of a decode step that has a 4.8 ms floor for an 8B model on an H100.

After the test

Picking an option is easier than deriving the answer at a whiteboard. These pages make you derive it: