Walk one request through vLLM V1, from the HTTP call to the streamed token. Which process owns each step, and why is it split that way?

TL;DR: The API-server process tokenizes and detokenizes; a separate EngineCore process runs a busy loop of schedule, execute, update; one worker process per GPU runs the model. The scheduler has no prefill or decode phase: every step it hands each request a number of tokens so its computed count catches up with its known tokens, under one token budget (8,192 for the OpenAI server on an 80 GB H100 in v0.31). The split exists so that Python work on the CPU overlaps with GPU work instead of sitting between kernels.

How to approach it

Say which version you are describing and that you are describing the V1 engine, because V0 internals (the block manager, the separate prefill and decode passes) are still what most blog posts show. Then walk the request in order and, at each stage, name the process, the data structure handed to the next stage, and what would stall the GPU if that stage were slow. Ask whether the interviewer wants tensor parallelism in the picture, since it adds one worker per GPU.

A strong answer

Described against the v0.31.0 source (October 2026):

1. API server. AsyncLLM receives the OpenAI-style request. Its InputProcessor applies the chat template, tokenizes, validates sampling parameters and builds an engine request. Doing this here, not in the engine, means a 100K-token prompt being tokenized never delays a decode step that is already running.
2. Into the engine. The request crosses to the EngineCore process over ZMQ, serialized with msgpack. EngineCore owns the scheduler and the KV cache manager, and runs a loop that does three things: drain its input queue, call scheduler.schedule(), run the step on the executor, then apply results with update_from_output.
3. Scheduling. The source states the rule directly: there is no prefill phase and no decode phase. Each request has a count of tokens computed and a count of tokens known (prompt plus outputs plus any speculative drafts), and each step assigns it tokens to close the gap, from a shared budget of max_num_batched_tokens. Running requests are served first, then waiting ones. A new prompt larger than the remaining budget is simply chunked, which is why chunked prefill is not a special mode in V1 but the default behaviour. Prefix caching sits inside the same step: before allocating, the KV cache manager looks up hashes of the prompt's full blocks, and a hit advances the computed count without scheduling those tokens at all.
4. Memory. If a request needs blocks the pool cannot supply, the scheduler preempts the lowest-priority running request, which under the default FCFS policy is the most recently admitted, frees its blocks, and moves it back to waiting for recompute.
5. Execution. The executor broadcasts the SchedulerOutput (which requests, how many tokens each, which blocks) to every worker through a shared-memory message queue. Each worker's model runner builds the flattened batch and its attention metadata, replays a captured CUDA graph when the batch shape allows, and samples. Only one worker's sampled ids go back to EngineCore, since every rank computed the same tokens.
6. Back out. EngineCore appends the tokens, checks stop conditions, and ships EngineCoreOutputs to the API server, whose OutputProcessor detokenizes incrementally and streams. Detokenization also lives outside the engine for the same reason tokenization does.

Async scheduling is the last piece. With it, EngineCore schedules step $n+1$ while step $n$ is still on the GPU, using placeholders for tokens it has not seen yet. It is enabled by default in v0.31 unless an incompatible option is set, such as pipeline parallelism on the V1 model runner, a pooling model, or an unsupported speculative method. The model runner itself was rewritten: in v0.31 the engine selects Model Runner V2 by default and falls back to the original runner only when a requested feature is not yet supported there (several speculative methods among them), with VLLM_USE_V2_MODEL_RUNNER as the override. Its README still carries an "experimental" label, which is exactly the kind of stale signal to check against the selection logic rather than trust.

Why the split matters, in numbers: a decode step for an 8B model on an H100 is a few milliseconds. If scheduling, tokenization and detokenization cost even 1 ms of Python each on the same thread, the GPU idles for a third of every step. Separate processes and an async scheduler turn that serial cost into overlapped cost.

The reversal condition: with very large models, where one step is tens of milliseconds, host overhead is a small fraction and the split buys less. It matters most for small, fast models at high request rates.

What interviewers probe next

"Where do the defaults come from?" max_num_batched_tokens and max_num_seqs depend on device memory and entry point: on an 80 GB H100 the OpenAI server defaults to 8,192 tokens and 1,024 sequences, and on parts with at least 160 GiB of memory, such as the 180 GB B200, the token budget rises to 16,384. An H200's 141 GB is only 131 GiB, so it gets the H100 defaults.

"What changes under tensor parallelism?" One worker per GPU, all receiving the same scheduler output; all-reduces happen inside the forward pass over NCCL or a custom all-reduce, and only one rank's samples return.

"A client disconnects mid-stream." The API server aborts the request in its output processor and in EngineCore; the scheduler frees its blocks at the next step, and any full blocks stay hashed for later prefix hits.

How it's scored

Weak: describes one Python process with a prefill queue and a decode queue, or places the block manager on the GPU workers.

Hire: separates the API server, EngineCore and workers, explains the token-budget scheduler with chunked prefill and prefix caching inside it, and names preemption.

Strong Hire: explains why tokenization and detokenization sit outside the engine, describes async scheduling and when it turns itself off, and quotes defaults with their version and hardware dependence.

Common mistakes

• Describing V0's separate prefill and decode passes as current behaviour.
• Treating max_num_seqs as the concurrency limit. With long contexts the KV pool binds first.
• Ignoring that defaults differ between the LLM class and the OpenAI server: on an 80 GB H100 the offline class gets twice the token budget.
• Putting detokenization on the critical path in a design discussion, then wondering why small models stream slowly.