Question 1

Walk one request through vLLM V1, from the HTTP call to the streamed token. Which process owns each step, and why is it split that way?

Accepted Answer

Five processes, one scheduler with no prefill phase, and a busy loop that should never wait on Python. The answer maps each stage to the component that owns it in the current source, not the 2023 paper.

Question 2

What do gpu_memory_utilization, max_model_len and max_num_seqs each control in vLLM, and how do they interact at startup?

Accepted Answer

Three flags, three different budgets, and one startup check that fails a 70B deployment on two H100s with a message most people misread. The answer walks the memory profile in the order vLLM runs it.

Question 3

Why do serving engines capture a CUDA graph per batch size, and what does padding to the nearest captured size actually cost?

Accepted Answer

A graph freezes shapes, so engines capture dozens of them and round every batch up. The answer prices that rounding in the regime where decode lives and finds the real costs somewhere else.

Question 4

Walk through how vLLM's prefix caching identifies a reusable KV block and hands it to a new request, step by step.

Accepted Answer

A block's identity is a hash of its tokens and of every block before it. The answer traces one request through hashing, lookup, the last-token rule and eviction, with a runnable model of the lookup.

Question 5

SGLang's radix cache versus vLLM's block hashing: compare the data structures, the eviction, and the workloads where each one wins.

Accepted Answer

Both reuse KV for shared prefixes, and on a 2K-token chat prompt their hit rates differ by under one percent. The answer finds the differences that do matter, from eviction policy to whether a cache key means anything outside one process.

Question 6

What does SGLang's overlap scheduler hide, how does it build the next batch before the current tokens exist, and what does it cost?

Accepted Answer

The GPU finishes a decode step and waits while Python appends tokens, checks stop strings and plans the next batch. The answer shows how the loop reorders that work, where the placeholder tokens live, and the request that pays for it.

Question 7

TensorRT-LLM in 2026: what is the PyTorch backend, how does its in-flight batching admit requests, and when do you still pick it?

Accepted Answer

Most answers still describe building a TensorRT engine per GPU and batch shape. The current release accepts only the PyTorch backend, and its default admission policy can halve your concurrency. The answer covers both, then says when it is still the right engine.

Question 8

Why is streaming detokenization harder than decoding each token, and what breaks with multi-byte characters and stop strings?

Accepted Answer

Decode each token on its own and an emoji streams as two replacement characters. The answer builds the offset-window decoder that fixes it, runs it, and adds the stop-string holdback that most streaming bugs come from.

Question 9

How does an engine batch requests for different LoRA adapters in one forward pass, and what do Punica-style shrink and expand kernels do?

Accepted Answer

Eight tenants, eight adapters, one base model and one batch. The answer explains why the extra cost scales with distinct adapters rather than requests, and which default serializes them.

Question 10

With JSON-schema decoding on, where in the engine step is the grammar mask computed and applied, and what does it cost?

Accepted Answer

The mask is a 16 KB bitmap per request. Whether it costs nothing or stalls every step depends on which thread computes it and what it is allowed to overlap with.

Question 11

You upgraded the serving engine with the same flags and throughput fell. Which defaults do you diff first, and how do you prove it?

Accepted Answer

Same command line, new version, lower throughput. The answer is a method: capture the resolved configuration on both sides, diff the handful of defaults that move every quarter, then pin and bisect.

Question 12

llama.cpp versus vLLM: what is each engine built for, and when does the CPU or Apple Silicon path actually win?

Accepted Answer

One engine is built to run a model anywhere with one user; the other is built to share a datacenter GPU across hundreds. The answer turns that into memory-bandwidth arithmetic and finds the cases where the laptop wins.

Question 13

Write the main loop of a minimal LLM serving engine: request queue, scheduler, model step, sampling and streaming.

Accepted Answer

About a hundred lines of Python that admit, chunk, preempt, sample and stream. The interviewer is watching where the KV accounting lives and what happens when the pool runs dry.

Question 14

What changed in vLLM's Model Runner V2 compared with the V1 runner, how does the engine decide which one to use, and what falls back?

Accepted Answer

The scheduler did not change; the code that turns its decisions into GPU tensors did. The answer covers where per-request state now lives, why Triton became a hard dependency, and the feature list that silently puts you back on the old runner.

Question 15

You are serving a reasoning model whose answers run 30K output tokens. Which engine do you pick, and how do you configure it?

Accepted Answer

Short prompts, enormous replies, and a KV footprint that grows every step after admission. The answer sizes the batch from the pool, prices a preemption, and shows why per-user speed, not throughput, is the number to defend.

Question 16

More CUDA graph captures made decode faster but increased preemption. Diagnose the regression and choose a policy.

Accepted Answer

Compare three capture policies using a memory ledger and a matched workload. Recover the missing KV pages, then defend a policy against both capacity and latency constraints.

Serving Engine Internals

vLLM, SGLang & TensorRT-LLM Interview Questions