Red Hat Inference Engineer interview questions

Per token, moving KV over PCIe is several times cheaper than recomputing it, yet vLLM recomputes. The answer derives the break-even bandwidth, shows how small copies fall below it, and names what makes recompute nearly free.

Chunked prefill keeps the decoders running, so the damage looks bounded. The walkthrough shows the inter-token gap climbing from 10 ms to a full second, new arrivals locked out for 8.6 seconds, and a slowdown that outlives the prefill.

The default of 1,024 sequences is never reached by a 2K-context workload on an 8B model, and is past the compute ridge for a 256-token one. The answer finds the three ceilings and picks the lowest.

Scheduling step n+1 before step n's tokens exist sounds impossible. The answer shows the placeholder trick that makes it work, what it costs at an end-of-sequence token, and the configurations where vLLM switches it off for you.

The indirection adds 4 bytes per 64 KiB of KV, so the byte count barely moves. The answer shows where the real cost hides: dependent loads, tiles that span several pages, and copy engines that want one contiguous box.

A 4-bit weight is only cheaper if turning it back into FP16 costs less than the bytes it saved. The answer counts the instruction budget per weight at full HBM bandwidth and shows the bit trick that makes the conversion fit.

Weight-only INT4 shrinks the bytes but keeps the arithmetic in BF16. The answer finds the batch size where that bargain flips, which does not depend on the model, then explains why a code benchmark notices before perplexity does.

Finer scales buy accuracy, and the bytes they cost are easy to count. The answer counts them for a 70B model, then shows the cost that matters more: which scales can wait for the epilogue and which ones invade the inner loop.

The two methods ship identical file formats and run on identical kernels, so the difference is entirely in how the rounding was chosen. The answer shows the one design choice that makes one of them fragile when traffic stops looking like the calibration set.

A handful of channels with huge activations wreck the INT8 grid for every other channel in the token. The answer shows why the obvious fix cannot run on a tensor core, what SmoothQuant does instead, and why FP8 makes most of the problem disappear.

Perplexity averages over the tokens a model finds easy and scores it on text it did not write. The answer builds the evaluation that catches what quantization actually breaks, and shows why a one-point drop on a benchmark can be noise or a real regression.

A few matrices cause most of the damage when a model is quantized, and keeping them wide costs little memory. The answer builds the sweep that finds them, prices it in GPU hours, and explains why the sum of single-layer damages is not the damage of the mix.

Local-inference formats spend every bit where it buys the most accuracy, and datacenter formats spend them uniformly. The answer works out what a Q4_K_M file actually costs per weight, and explains why the trade that wins on one consumer GPU loses at batch 64.

FP8 halves the weights and still leaves almost no memory for the cache. The answer does the budget line by line, picks the format pair that leaves room for real concurrency, and says which hardware change makes the opposite choice correct.

Four beams over a 1,000-token prompt cost 600 MiB if each keeps its own cache and as little as 156 MiB if they share blocks. The answer implements the fork, the copy-on-write and the prune, then shows what vLLM actually does today.

Five processes, one scheduler with no prefill phase, and a busy loop that should never wait on Python. The answer maps each stage to the component that owns it in the current source, not the 2023 paper.

Three flags, three different budgets, and one startup check that fails a 70B deployment on two H100s with a message most people misread. The answer walks the memory profile in the order vLLM runs it.

A block's identity is a hash of its tokens and of every block before it. The answer traces one request through hashing, lookup, the last-token rule and eviction, with a runnable model of the lookup.

Decode each token on its own and an emoji streams as two replacement characters. The answer builds the offset-window decoder that fixes it, runs it, and adds the stop-string holdback that most streaming bugs come from.

The mask is a 16 KB bitmap per request. Whether it costs nothing or stalls every step depends on which thread computes it and what it is allowed to overlap with.

Same command line, new version, lower throughput. The answer is a method: capture the resolved configuration on both sides, diff the handful of defaults that move every quarter, then pin and bisect.

One engine is built to run a model anywhere with one user; the other is built to share a datacenter GPU across hundreds. The answer turns that into memory-bandwidth arithmetic and finds the cases where the laptop wins.

The scheduler did not change; the code that turns its decisions into GPU tensors did. The answer covers where per-request state now lives, why Triton became a hard dependency, and the feature list that silently puts you back on the old runner.

Sending every request with the same prefix to the same replica halves the fleet, until one customer's prompt gets popular. The answer derives the spill threshold from the prefill a cache hit saves, in milliseconds.

141 GB of weights on 160 GB of GPUs, and the engine refuses to start. The answer walks the five phases of engine startup, shows which one failed from the log, and does the arithmetic that says the model never fit the way it was configured.

FP8 halves the weight bytes, and the dashboard says the replica got slower. The answer computes what FP8 should have bought, splits the gap into kernel, scale and config terms with one profiler trace, and catches the regression that is not in the GPU at all.

The KV pool is preallocated, so the growth is somewhere the engine does not budget. The answer splits memory into what is fixed and what can grow, turns a growth rate into bytes per request, and uses that one number to rule out half the suspects.

The formula takes one line. The answer interviewers score is the memory budget around it: what the engine reserves first, what tensor parallelism does to the heads, and why the count you get is a ceiling.

The bound is a one-line argument about where a sequence can be partly full. The block size is a three-way trade between that waste, kernel efficiency and how finely the prefix cache can match.

Weights and KV are the two numbers everyone knows. The answer that gets the offer accounts for the other few gigabytes, says which ones vLLM measures and which it guesses, and knows where an OOM comes from when the sum looks fine.

FP8 halves the cache and the attention bytes, and on short prompts it looks free. The answer explains why the scale matters less than people think for rounding and more than they think for clipping, and why the failures hide at 128K.

Doubling the GPUs past the KV head count halves the weights per GPU and leaves the KV per GPU exactly where it was. The answer derives that, prices the duplicate copy, and picks the parallelism that does not pay it.

Half the layers capped at a 128-token window sounds like a big saving at 128K. The arithmetic says the saving is set by the layer ratio, not the window, and the allocator decides whether you get it at all.

Recomputing a prefix costs FLOPs proportional to the model, fetching it costs bytes proportional to the cache. One ratio decides it, and the answer shows by how much fetching wins and the three situations where it still loses.

A prefix cache only hits from the first token forward, so evicting the wrong block can strand gigabytes that stay resident and can never be hit. The answer derives how much, and which eviction order guarantees it never happens.