The vLLM V1 Engine Architecture

TL;DR: vLLM V1 runs three kinds of process: an API server that tokenizes and detokenizes, one EngineCore per data-parallel rank that owns the scheduler and the KV block pool, and one worker per GPU that runs the model. The split exists because a Llama 3.1 8B decode step on an H100 has a 4.8 ms floor, so every millisecond of Python left on the critical path costs about a fifth of the GPU. Described against v0.31.0, checked October 2026.

The processes and what crosses between them

Each boundary carries a small, explicit message. The API server sends token ids and sampling parameters, never text. The scheduler sends workers a SchedulerOutput: which requests run, how many tokens each, which new block ids each got. Workers keep their own persistent copy of batch state and apply that diff rather than rebuilding the batch. Workers return sampled ids, plus logprobs when asked. Detokenized text never enters EngineCore.

EngineCore: one loop, three steps

run_busy_loop polls the input queue, then calls a step: schedule, execute, update. Schedule produces the SchedulerOutput. Execute hands it to the executor, which broadcasts to every worker. Update, update_from_output, appends sampled tokens, checks stop conditions, frees finished requests' blocks and emits outputs.

The scheduler has no prefill or decode phase. Every request carries a count of tokens computed and a count of tokens known, and each step assigns tokens to close the gap under max_num_batched_tokens, running requests first, then waiting ones. A long prompt that does not fit the remaining budget gets the remainder, so chunked prefill is simply what the rule does. If a running request needs a block the pool cannot give, the scheduler preempts from the tail of the running list (the most recently admitted request under FCFS), frees its blocks, and returns it to waiting for recompute.

With async scheduling, on by default in v0.31.0 unless an incompatible option is set, EngineCore keeps a batch queue of two: it schedules step $n+1$ while step $n$ is still on the GPU, using placeholders for the token step $n$ has not produced yet.

The KV cache manager

The BlockPool owns every physical block. Free blocks sit in a doubly linked queue in eviction order, and a hash map from block hash to block makes prefix hits a lookup. A full block's hash chains its parent's hash with its own token ids and any extra keys (LoRA id, multimodal hashes), using SHA-256 by default, so equal hashes mean equal prefixes, not merely equal blocks. A freed block keeps its hash and goes to the queue's tail; it is reused from the head, which makes eviction least-recently-freed. A block that is later hit is pulled out of the free queue and its reference count goes up.

Sizing the pool for Llama 3.1 8B on an 80 GB H100 at the 0.92 default:

KV per token   = 2 x 32 layers x 8 kv_heads x 128 x 2 B = 128 KiB
block (16 tok) = 16 x 128 KiB = 2 MiB
budget         = 80 GB x 0.92 = 73.6 GB
minus weights  = 73.6 - 16.06 = 57.5 GB, before the profiled activation peak
blocks         <= 57.5e9 / 2,097,152 = 27,436 blocks = 439K tokens
at 4,096 tokens per sequence: <= 107 sequences resident
sanity: max_num_seqs defaults to 1,024 here, so the KV pool binds first

The worker and model runner

Each worker owns one GPU, the weights for its shard and the KV tensors. In v0.31.0 the engine selects Model Runner V2 by default and falls back to the original runner for features V2 does not support yet, such as several speculative methods; VLLM_USE_V2_MODEL_RUNNER overrides the choice. CUDA graphs default to FULL_AND_PIECEWISE: full graphs for uniform decode batches, piecewise graphs around attention for mixed ones. Capture sizes follow [1, 2, 4], then steps of 8 to 248, then steps of 16 to 512, which is 51 sizes; a batch of 37 runs padded in the 40 graph.

Where it breaks

• EngineCore's loop is one Python thread. At high request rates with a small model, scheduling and output handling can exceed a 5 ms step even with async scheduling. Data parallelism adds EngineCores; extra API-server processes split tokenization load.
• Placeholders limit async. Anything that needs the previous token on the host before the next schedule, such as some speculative methods, turns async scheduling off.
• Defaults are per hardware and entry point. The OpenAI server and the offline LLM class get different token budgets on the same GPU.

What interviewers listen for

• Process boundaries named with what crosses them.
• The scheduler's single rule, with chunked prefill and prefix hits falling out of it.
• Block-pool sizing from bytes per token, and the observation that KV binds before max_num_seqs.
• Version stated: Model Runner V2 and async scheduling are both defaults in v0.31.0.

Say it in 30 seconds

vLLM V1 puts tokenization and detokenization in API-server processes, the scheduler and KV block pool in an EngineCore process per DP rank, and the model in one worker per GPU. The scheduler has one rule: give each request tokens until its computed count reaches its known count, under one budget. The block pool hashes full 16-token blocks in a chain for prefix hits and evicts least-recently-freed. On an H100 an 8B model gets at most about 27K blocks, 439K tokens. The split keeps Python off a 4.8 ms step.