Perplexity Inference Engineer interview questions

One division tells you whether a headline number is physics or marketing. The answer runs it, then works out which unstated condition would make the claim true and what to ask the vendor for.

The crossover falls out of one division, and the number it gives lines up exactly with something else on the GPU. Spotting that coincidence is what the interviewer is waiting for.

A 2.5x gap at batch 1 almost never comes from the GPU being slow. The answer sorts the suspects by what a single profiler trace shows and puts a millisecond figure on each.

Recomputing a prefix costs FLOPs proportional to the model, fetching it costs bytes proportional to the cache. One ratio decides it, and the answer shows by how much fetching wins and the three situations where it still loses.

A prefix cache only hits from the first token forward, so evicting the wrong block can strand gigabytes that stay resident and can never be hit. The answer derives how much, and which eviction order guarantees it never happens.

Absorption is one line of matrix associativity. The part interviewers score is the bill: attention gets more expensive per cached token, and the kernel lands in a roofline regime GQA never visits.

All three policies keep the server alive. They differ in who waits, who redoes work, and which percentile absorbs the damage, and the numbers for a 6K-token victim make the choice for you.

The transfer looks like gigabytes per request until you divide by prefill time. Then the prompt length cancels, and the real question becomes which models and which GPUs break the link.

Each tier is worth exactly the recompute it avoids minus the fetch it costs. Work that out per token and the design writes itself, including the tier that should not exist for small models and the SSD that wears out in a week.

One 32K-token prompt in front of ten short ones costs each of them more than a second of TTFT. The answer separates the half of the job length you know from the half you do not, and schedules each differently.

The 70% rule of thumb falls out of one formula once you plug in a prefill time and a p99 target. The answer derives it, shows how prompt-length variance moves it to 50%, and lists the four ways an LLM engine breaks the textbook queue.

Chunked prefill keeps the decoders running, so the damage looks bounded. The walkthrough shows the inter-token gap climbing from 10 ms to a full second, new arrivals locked out for 8.6 seconds, and a slowdown that outlives the prefill.

Accepting everything during a one-minute spike leaves the API serving dead requests for two minutes after it ends. The answer does the queue arithmetic, picks the rejection rule, and writes the response that keeps clients from making it worse.

Batch-1 decode at long context gives the GPU eight thread blocks and 132 SMs. The answer counts the waves, picks the split count from a wave-efficiency rule rather than a guess, and derives the merge that makes the result exact.

Fusing RoPE saves a read and a write of Q and K, which is real but small in prefill. The precision bug is not small: one tensor in the wrong dtype scrambles positional information by a few hundred tokens. The answer sizes both.

CUDA lets you reserve part of L2 for data you choose, and the KV cache is re-read every step. The answer measures what fits against what a decode step streams, finds the one case where pinning pays, and the kernel change that pays more.

Fine-grained scales let E4M3 carry every tensor, but they change along the reduction axis, so the tensor cores cannot apply them. The answer walks the main loop where they do get applied, prices that work against the MMA, and explains what Blackwell moves into hardware.

Calibration data reaches each expert only through the router, so the experts that matter for rare domains see almost none of it. The answer counts the tokens, shows the point where a Hessian-based quantizer becomes worse than plain rounding, and builds a recipe that does not depend on luck.

DeepSeek V3 ships a drafter it trained alongside the model for free. The answer walks the MTP module's wiring and loss, prices a draft step against a 37.6B-active target, and reproduces the reported 1.8x from first principles.

Compute acceptance from counter snapshots, reject reset-contaminated windows, and handle EOS before inferring request-step counts. Then decide what a matched-load baseline must measure.

Every MoE layer sends each token out to its experts and gathers the results back, twice per layer, 58 times per step. The answer turns DeepSeek V3's config into bytes per token, splits them between NVLink and the NIC, and shows what node-aware dispatch saves.

Tensor parallelism splits a GQA cache by KV head. MLA has no KV heads to split, so TP copies the whole cache onto every GPU without any warning. The answer measures that waste on an H200 node and prices the fix.

An 8k-token prompt on a 70B model leaves 2.7 GB of KV that has to cross the network before decode can start. The answer shows that the sustained bandwidth is modest, that the latency is the real problem, and which side of the link runs out first.

Same eight GPUs, same weights per GPU, same KV capacity. One layout pays 252 all-reduces per token, the other pays for every layer twice in wall-clock time. The answer works both, then finds the precision choice that forces the second.

Eight GPUs per NVLink domain shaped every serving layout of the Hopper era. Seventy-two moves the boundary that expert parallelism and KV transfer used to cross over the NIC. The answer prices what moves inside the rack, what still does not fit, and the failure mode that comes with it.

Inside one batch there is nothing to overlap: every layer waits for its all-to-all. Split the batch in two and one half computes while the other communicates. The answer models the gain, finds the batch size where splitting makes things slower, and checks it against a shipped default.

A block's identity is a hash of its tokens and of every block before it. The answer traces one request through hashing, lookup, the last-token rule and eviction, with a runnable model of the lookup.

Most answers still describe building a TensorRT engine per GPU and batch shape. The current release accepts only the PyTorch backend, and its default admission policy can halve your concurrency. The answer covers both, then says when it is still the right engine.

About a hundred lines of Python that admit, chunk, preempt, sample and stream. The interviewer is watching where the KV accounting lives and what happens when the pool runs dry.

Short prompts, enormous replies, and a KV footprint that grows every step after admission. The answer sizes the batch from the pool, prices a preemption, and shows why per-user speed, not throughput, is the number to defend.

Most designs draw the router first. The answer that passes does the arithmetic first: tokens per second in each phase, replica-seconds per second, a utilization ceiling, then dollars, and finds that the prompt, not the reply, is two thirds of the bill.

Autoscaling cannot save a failover when a replica takes six minutes to come up, so the spare capacity has to exist before the region dies. The answer solves for the smallest fleet that survives any single region loss, and places the spare next to the region most likely to need it.

A canary is only as good as the number of requests behind each verdict. The answer sizes the canary from the statistics of the regression it must catch, then stages the rollout around that number.

Inspect tokenized prompts and cache-counter deltas to diagnose a TTFT regression. A runnable prefix-boundary check and a routing follow-up test whether the evidence actually supports the diagnosis.

Users see the first token quickly and then the stream stutters. The answer builds the decode step from bytes, shows which cause produces a bimodal latency histogram, and names the one metric that tells the top two apart.

Prompts fit, but their outputs exhaust the pool. Work through a page-level memory ledger, choose the next diagnostic check, and test a concurrency fix against queueing and goodput.

A regular period is a clue most candidates throw away. The answer turns it into a ranked list of periodic sources, shows why a few slow steps out of hundreds own the p99, and names the one correlation that separates the top two.

The speedup formula has two inputs that vary per request and per moment, and the tail lives where both are bad. The answer computes when speculation turns into a slowdown, shows which requests own the p99, and picks a policy instead of an on/off switch.

Matching a prefix is a trie walk. The grade is in the edge split, which cached tokens a running request pins, and an eviction loop that turns parents into leaves without ever freeing a slot twice.

Two prompts with the same 16 tokens in block 3 must not share block 3 unless blocks 0 to 2 match too. The answer chains the hashes, keeps freed blocks hittable until memory is needed, evicts tails before heads, and proves a colliding hash cannot hand a request the wrong KV.

The stop string arrives as three tokens, the euro sign arrives as two, and the client has already been sent half of both. The answer holds back exactly the text that could still become a stop, decodes bytes incrementally, and tests the stream against a reference that sees the whole output at once.

The formula takes one line. The answer interviewers score is the memory budget around it: what the engine reserves first, what tensor parallelism does to the heads, and why the count you get is a ceiling.

The bound is a one-line argument about where a sequence can be partly full. The block size is a three-way trade between that waste, kernel efficiency and how finely the prefix cache can match.

Weights and KV are the two numbers everyone knows. The answer that gets the offer accounts for the other few gigabytes, says which ones vLLM measures and which it guesses, and knows where an OOM comes from when the sum looks fine.

FP8 halves the cache and the attention bytes, and on short prompts it looks free. The answer explains why the scale matters less than people think for rounding and more than they think for clipping, and why the failures hide at 128K.

Doubling the GPUs past the KV head count halves the weights per GPU and leaves the KV per GPU exactly where it was. The answer derives that, prices the duplicate copy, and picks the parallelism that does not pay it.

Half the layers capped at a 128-token window sounds like a big saving at 128K. The arithmetic says the saving is set by the layer ratio, not the window, and the allocator decides whether you get it at all.

Count the blocks with and without sharing and the saving looks large. The answer that scores also knows when the copy fires, what a misaligned prompt does, and the workload where the saving all but disappears.

The per-token formula gives a scary number for GQA and a reassuring one for MLA. Put both on real GPUs with tensor parallelism and the reassuring one is the request that does not fit.