Baseten Inference Engineer interview questions

Decode and prefill run the same weights through the same kernels, yet one waits on memory and the other on tensor cores. The answer finds the prompt length where they trade places and turns it into a TTFT.

One division tells you whether a headline number is physics or marketing. The answer runs it, then works out which unstated condition would make the claim true and what to ask the vendor for.

The datasheet ratios are close to each other, so the lazy answer is one speedup for everything. The answer splits it into three regimes, and the biggest win turns out to come from capacity, not bandwidth.

MBU is one division, and that is why it is so easy to get wrong. The answer defines both operands precisely on a real server and then shows six ways a dashboard can report a number 30 points off.

A 60% MBU number is a symptom, not a diagnosis. The answer rebuilds the floor from bytes, then splits the missing 40% into buckets a profiler can confirm and a team can own.

Step time is a straight line in batch size, so throughput is a hyperbola with a ceiling. The answer finds the knee in closed form and shows why the ceiling usually comes from the KV cache, not the tensor cores.

Weights and KV are the two numbers everyone knows. The answer that gets the offer accounts for the other few gigabytes, says which ones vLLM measures and which it guesses, and knows where an OOM comes from when the sum looks fine.

Recomputing a prefix costs FLOPs proportional to the model, fetching it costs bytes proportional to the cache. One ratio decides it, and the answer shows by how much fetching wins and the three situations where it still loses.

Count the blocks with and without sharing and the saving looks large. The answer that scores also knows when the copy fires, what a misaligned prompt does, and the workload where the saving all but disappears.

All three policies keep the server alive. They differ in who waits, who redoes work, and which percentile absorbs the damage, and the numbers for a 6K-token victim make the choice for you.

The transfer looks like gigabytes per request until you divide by prefill time. Then the prompt length cancels, and the real question becomes which models and which GPUs break the link.

The gain is not a constant. It is a statistic of your output-length distribution, and the right token budget is a point on the roofline. The answer derives both, then says where the paper number and the real knee disagree.

Doubling the batch raised tokens per second by a third and sent goodput to zero. The answer shows why every sequence in a batch shares one step time, and how to find the batch size that maximizes the number users feel.

One multiplication gives the number of sequences resident on the GPU, and from it the KV bytes and the batch the step runs at. The trap is that the duration in that multiplication depends on the answer.

Request-fair scheduling hands 96% of the GPU to the tenant with long documents. The answer derives the token weights from the roofline, shows the counter that enforces them per step, and the one rule that stops a tenant from banking credit.

The default of 1,024 sequences is never reached by a 2K-context workload on an 8B model, and is past the compute ridge for a 256-token one. The answer finds the three ceilings and picks the lowest.

Batch-1 decode at long context gives the GPU eight thread blocks and 132 SMs. The answer counts the waves, picks the split count from a wave-efficiency rule rather than a guess, and derives the merge that makes the result exact.

The indirection adds 4 bytes per 64 KiB of KV, so the byte count barely moves. The answer shows where the real cost hides: dependent loads, tiles that span several pages, and copy engines that want one contiguous box.

Eight tiles too many turns a one-wave GEMM into two. The answer prices the waste in microseconds for a real prefill shape, then compares the three fixes, including the one that lives in the scheduler, not the kernel.

Every kernel in the step can run near peak bandwidth while the step itself runs at two thirds of it. The answer reads the three rows of the timeline that tell you why, and turns the gaps into milliseconds.

On paper an H200 SXM is an H100 with more and faster memory, so nothing should get slower. The answer works through the four ways it can anyway, from the spec sheet you did not read to the tuning table that does not know the new device's name.

The two methods ship identical file formats and run on identical kernels, so the difference is entirely in how the rounding was chosen. The answer shows the one design choice that makes one of them fragile when traffic stops looking like the calibration set.

Both formats store the same eight 4-bit values per element. The difference is all in the scale, and the answer shows which of the two scale choices buys more accuracy and what it costs in bits.

Local-inference formats spend every bit where it buys the most accuracy, and datacenter formats spend them uniformly. The answer works out what a Q4_K_M file actually costs per weight, and explains why the trade that wins on one consumer GPU loses at batch 64.

FP8 halves the weights and still leaves almost no memory for the cache. The answer does the budget line by line, picks the format pair that leaves room for real concurrency, and says which hardware change makes the opposite choice correct.

At temperature zero, verification is a prefix match, and the whole implementation fits in a dozen vectorized lines. Interviewers grade the three details around it: the bonus token, the KV slots to roll back, and why the output can still differ from plain greedy.

One option drafts four tokens for six percent of a target step but guesses each position blind; the other conditions every guess and costs nearly half a step. The answer prices both on Llama 3.3 70B and finds the batch size where the winner flips.

DeepSeek V3 ships a drafter it trained alongside the model for free. The answer walks the MTP module's wiring and loss, prices a draft step against a 37.6B-active target, and reproduces the reported 1.8x from first principles.

Compute acceptance from counter snapshots, reject reset-contaminated windows, and handle EOS before inferring request-step counts. Then decide what a matched-load baseline must measure.

A per-request schema compile stalls the scheduler, a per-step mask must beat the forward pass, and skipping forced characters can push the model off its own tokenization. The answer handles all three with numbers from one realistic schema.

The same drafter that triples speed at 3 a.m. costs a third of peak throughput at 6 p.m. if left alone. The answer builds a step-time model for Llama 3.3 70B on eight H100s, derives a draft length per batch bucket, and shows why context length moves the whole table.

Tensor parallelism divides the weight bytes but not the all-reduce count, so each doubling buys less than the one before. The answer prices that fixed cost, then finds the workload where TP=8 wins anyway.

Two matrix multiplies in a row, and the order of the splits decides whether you communicate once or twice per block. The answer places every split for a 70B layer and prices what the remaining all-reduces send per token.

Tensor parallelism splits a GQA cache by KV head. MLA has no KV heads to split, so TP copies the whole cache onto every GPU without any warning. The answer measures that waste on an H200 node and prices the fix.

An 8k-token prompt on a 70B model leaves 2.7 GB of KV that has to cross the network before decode can start. The answer shows that the sustained bandwidth is modest, that the latency is the real problem, and which side of the link runs out first.

A ring all-reduce is bandwidth-optimal and takes fourteen synchronized steps on eight GPUs. At 16 KB per call, the steps are the cost. The answer builds the latency-bandwidth model, finds the crossover size, and checks it against the threshold vLLM ships.

Collectives run at the speed of their slowest link, so one NIC that trained at a quarter of its rate slows every GPU in the instance by the same amount. The answer shows the arithmetic that makes a 4x slower link a 2x slower step, then bisects the cluster to the culprit.

Pulling every replica's copy from object storage turns a 500 GB model into 62.5 TB of reads. The answer finds the three bounds that actually limit a fleet load, shows a pipelined broadcast reaching the tightest one, and lists the startup work that hides behind the bytes.

Three flags, three different budgets, and one startup check that fails a 70B deployment on two H100s with a message most people misread. The answer walks the memory profile in the order vLLM runs it.

A graph freezes shapes, so engines capture dozens of them and round every batch up. The answer prices that rounding in the regime where decode lives and finds the real costs somewhere else.

Most answers still describe building a TensorRT engine per GPU and batch shape. The current release accepts only the PyTorch backend, and its default admission policy can halve your concurrency. The answer covers both, then says when it is still the right engine.

Eight tenants, eight adapters, one base model and one batch. The answer explains why the extra cost scales with distinct adapters rather than requests, and which default serializes them.

Same command line, new version, lower throughput. The answer is a method: capture the resolved configuration on both sides, diff the handful of defaults that move every quarter, then pin and bisect.

Compare three capture policies using a memory ledger and a matched workload. Recover the missing KV pages, then defend a policy against both capacity and latency constraints.

Most designs draw the router first. The answer that passes does the arithmetic first: tokens per second in each phase, replica-seconds per second, a utilization ceiling, then dollars, and finds that the prompt, not the reply, is two thirds of the bill.

Four boxes are easy to draw. The answer that passes sizes them first, then says exactly which clock starts where, because a TTFT measured inside the engine and a TTFT measured by the client can disagree by a factor of five.

A web-service autoscaler watching GPU utilization will scale this fleet too late and in the wrong direction. The answer turns the four-minute load time into a headroom number, then decides what absorbs the bursts no scaler can catch.

Five hundred merged fine-tunes need five hundred GPUs before the first request. Served as adapters on a shared base they need a few dozen. The answer sizes the adapters, places them in a three-tier memory hierarchy, and finds the one setting that silently multiplies their memory by eight.

Twenty models with two replicas each is a fleet that is mostly idle. The answer measures how idle, then tiers the models by load so the long tail shares GPUs without hurting the models that pay the bills.

Input and output tokens burn different resources, so they get different prices. The answer prices each phase from first principles, then shows which assumption moves the list price most.

A cold 32K prefill on a 70B model barely fits a second of compute on eight H100s, before any queueing. The design that passes makes almost every request warm, then sizes two pools from what remains.

Voice moves the latency target from the first token to the first audible clause, and the largest item in the budget is not a model at all. The answer allocates 300 ms line by line, then sizes the LLM tier from it.

A cheaper GPU and a faster decoder do not automatically produce a cheaper service. Audit a cost ledger, reject misleading wins, and choose a rollout only after quality and latency gates pass.

Users see the first token quickly and then the stream stutters. The answer builds the decode step from bytes, shows which cause produces a bimodal latency histogram, and names the one metric that tells the top two apart.

141 GB of weights on 160 GB of GPUs, and the engine refuses to start. The answer walks the five phases of engine startup, shows which one failed from the log, and does the arithmetic that says the model never fit the way it was configured.

FP8 halves the weight bytes, and the dashboard says the replica got slower. The answer computes what FP8 should have bought, splits the gap into kernel, scale and config terms with one profiler trace, and catches the regression that is not in the GPU at all.

Same model, same image, same instance type, and one replica drags the fleet's p99. The answer first proves the replica is slower at equal load rather than busier, then walks the hardware causes from GPU clocks to a link that trained at the wrong speed.

Both numbers can be true at once. The answer shows how to turn the gap into a short list of measurable dilutions, which one usually eats most of the 2x, and the replay test that settles it.

A regular period is a clue most candidates throw away. The answer turns it into a ranked list of periodic sources, shows why a few slow steps out of hundreds own the p99, and names the one correlation that separates the top two.

Small models make the GPU step so short that host work stops hiding behind it. The answer shows how to prove that with a timeline instead of a hunch, how much throughput a few milliseconds of Python cost, and which fixes help in which order.

The KV pool is preallocated, so the growth is somewhere the engine does not budget. The answer splits memory into what is fixed and what can grow, turns a growth rate into bytes per request, and uses that one number to rule out half the suspects.

A dashboard of forty graphs is a dashboard nobody reads at 2 a.m. The answer builds eight panels in the order an incident unfolds, ties each to the action it decides, and shows how two of them check each other with Little's law.

Use cold-schema and warm-batch traces to separate compilation from recurring stalls. Identify the exposed delay, choose a targeted experiment, and preserve output correctness.

Reconstruct token arrival from a request-correlated trace. Separate transfer overlap from decode admission, and show how first-token delivery changes the metric that exposes a handoff stall.

Unused logical KV pages can coexist with exhausted device memory. Reconcile expanded image tokens, retained embeddings and phase-specific peaks before changing a decoder token limit.

Replay the same burst through affinity, round-robin and predicted-finish routing. Then remove local reservations or delay an eviction notice and explain exactly why the decisions change.

Everyone can sort and cumsum. The grader feeds you tied logits, p = 1 on a 128K vocabulary and a temperature of zero, and checks whether your kept set depends on the order of the sort.

Matching a prefix is a trie walk. The grade is in the edge split, which cached tokens a running request pins, and an eviction loop that turns parents into leaves without ever freeing a slot twice.

A request costs anything from ten tokens to a hundred thousand, and you only learn the real cost when it finishes. The answer reserves at admission, refunds at completion, fails fast on requests that can never fit, and tests the bound the bucket is supposed to guarantee.

The stop string arrives as three tokens, the euro sign arrives as two, and the client has already been sent half of both. The answer holds back exactly the text that could still become a stop, decodes bytes incrementally, and tests the stream against a reference that sees the whole output at once.

A disconnect arrives after launch, then the request ID is reused. Implement deferred reclamation and prove that late or duplicate completions cannot emit stale output or free another request's pages.

Accepted, emitted and computed tokens are different counts. Trace rejection, full acceptance, EOS and an output cap to find the state that the next forward pass is allowed to read.

One slow connection can stall unrelated generations or retain an unbounded output queue. Implement a byte budget that includes the pending write, then connect overflow and disconnects to engine cancellation.

The formula takes one line. The answer interviewers score is the memory budget around it: what the engine reserves first, what tensor parallelism does to the heads, and why the count you get is a ceiling.

The bound is a one-line argument about where a sequence can be partly full. The block size is a three-way trade between that waste, kernel efficiency and how finely the prefix cache can match.

FP8 halves the cache and the attention bytes, and on short prompts it looks free. The answer explains why the scale matters less than people think for rounding and more than they think for clipping, and why the failures hide at 128K.

Doubling the GPUs past the KV head count halves the weights per GPU and leaves the KV per GPU exactly where it was. The answer derives that, prices the duplicate copy, and picks the parallelism that does not pay it.

Half the layers capped at a 128-token window sounds like a big saving at 128K. The arithmetic says the saving is set by the layer ratio, not the window, and the allocator decides whether you get it at all.

A prefix cache only hits from the first token forward, so evicting the wrong block can strand gigabytes that stay resident and can never be hit. The answer derives how much, and which eviction order guarantees it never happens.

Absorption is one line of matrix associativity. The part interviewers score is the bill: attention gets more expensive per cached token, and the kernel lands in a roofline regime GQA never visits.

The per-token formula gives a scary number for GQA and a reassuring one for MLA. Put both on real GPUs with tensor parallelism and the reassuring one is the request that does not fit.