Anthropic Inference Engineer interview questions

Decode reads every weight once per token. The answer turns that sentence into a millisecond floor, then accounts for the three things that keep a real server above it.

Two FLOPs per parameter is the number everyone quotes. The answer interviewers want rebuilds it from the config, finds the parameters that do no arithmetic, and names the context length where the rule breaks.

Decode and prefill run the same weights through the same kernels, yet one waits on memory and the other on tensor cores. The answer finds the prompt length where they trade places and turns it into a TTFT.

Batching rescues the weight GEMMs at decode but never the attention kernel, and one head-count ratio explains why. The answer derives it, then shows how MLA moves attention to within a hair of the H100 ridge.

A 60% MBU number is a symptom, not a diagnosis. The answer rebuilds the floor from bytes, then splits the missing 40% into buckets a profiler can confirm and a team can own.

The crossover falls out of one division, and the number it gives lines up exactly with something else on the GPU. Spotting that coincidence is what the interviewer is waiting for.

A 2.5x gap at batch 1 almost never comes from the GPU being slow. The answer sorts the suspects by what a single profiler trace shows and puts a millisecond figure on each.

Every weight GEMM ties, attention sits at a constant the batch cannot move, and the ranking flips at batch 8. The answer then separates what fusion can fix from what only the model architecture or the batch can.

Twenty lines of roofline arithmetic get within striking distance of a real server, and the interviewer knows it. What they are scoring is the list of places the model breaks, and how you would catch each one before a capacity plan rests on it.

The formula takes one line. The answer interviewers score is the memory budget around it: what the engine reserves first, what tensor parallelism does to the heads, and why the count you get is a ceiling.

The bound is a one-line argument about where a sequence can be partly full. The block size is a three-way trade between that waste, kernel efficiency and how finely the prefix cache can match.

FP8 halves the cache and the attention bytes, and on short prompts it looks free. The answer explains why the scale matters less than people think for rounding and more than they think for clipping, and why the failures hide at 128K.

Recomputing a prefix costs FLOPs proportional to the model, fetching it costs bytes proportional to the cache. One ratio decides it, and the answer shows by how much fetching wins and the three situations where it still loses.

A prefix cache only hits from the first token forward, so evicting the wrong block can strand gigabytes that stay resident and can never be hit. The answer derives how much, and which eviction order guarantees it never happens.

Count the blocks with and without sharing and the saving looks large. The answer that scores also knows when the copy fires, what a misaligned prompt does, and the workload where the saving all but disappears.

All three policies keep the server alive. They differ in who waits, who redoes work, and which percentile absorbs the damage, and the numbers for a 6K-token victim make the choice for you.

The per-token formula gives a scary number for GQA and a reassuring one for MLA. Put both on real GPUs with tensor parallelism and the reassuring one is the request that does not fit.

Each tier is worth exactly the recompute it avoids minus the fetch it costs. Work that out per token and the design writes itself, including the tier that should not exist for small models and the SSD that wears out in a week.

The gain is not a constant. It is a statistic of your output-length distribution, and the right token budget is a point on the roofline. The answer derives both, then says where the paper number and the real knee disagree.

Doubling the batch raised tokens per second by a third and sent goodput to zero. The answer shows why every sequence in a batch shares one step time, and how to find the batch size that maximizes the number users feel.

One multiplication gives the number of sequences resident on the GPU, and from it the KV bytes and the batch the step runs at. The trap is that the duration in that multiplication depends on the answer.

One 32K-token prompt in front of ten short ones costs each of them more than a second of TTFT. The answer separates the half of the job length you know from the half you do not, and schedules each differently.

A priority flag fixes the queue and leaves the real damage in place: batch sequences that are already running slow every interactive token. The answer puts a number on the KV the batch class may hold.

The 70% rule of thumb falls out of one formula once you plug in a prefill time and a p99 target. The answer derives it, shows how prompt-length variance moves it to 50%, and lists the four ways an LLM engine breaks the textbook queue.

Chunked prefill keeps the decoders running, so the damage looks bounded. The walkthrough shows the inter-token gap climbing from 10 ms to a full second, new arrivals locked out for 8.6 seconds, and a slowdown that outlives the prefill.

The two targets are coupled: the TPOT target caps the prefill chunk, which caps prefill throughput, which caps the backlog the TTFT target can absorb. The answer builds the step model, derives the admission rule and shows what to do with requests that fail it.

Accepting everything during a one-minute spike leaves the API serving dead requests for two minutes after it ends. The answer does the queue arithmetic, picks the rejection rule, and writes the response that keeps clients from making it worse.

FlashAttention does the same arithmetic as naive attention and is several times faster. The answer is a three-line recurrence and one IO count, and interviewers want both.

The one-query recurrence is a warm-up. A real kernel carries a vector of maxima and a vector of sums for a tile of rows, and the rescale becomes a diagonal matrix. The answer writes it, runs it, and shows the edge case that turns it into NaN.

The textbook IO count says FlashAttention re-reads K and V once per query block, which at 8K is only 2x better than naive. The answer explains why the real HBM number is 65x lower, and which cache makes the difference.

Storing the attention matrix for backward costs gigabytes per layer; recomputing it costs extra FLOPs. The answer derives the gradient, shows the one scalar per row that makes recomputation exact, and the identity that removes the last N-squared term.

FlashAttention-2 reaches well under half of an H100's BF16 peak. The answer explains why in one ratio, exponentials against matmuls, and maps each FA3 technique to the stall it hides.

An error that grows with context length is a running sum losing increments, and BF16 stops adding 1 to a total at 256. The answer turns the symptom into a bisection, finds the state held in the wrong dtype, and shows the error curve that proves it.

At 1M tokens a 70B GQA model needs 320 GiB of KV per request and spends over 90% of prefill FLOPs in attention. The answer lays out a plan that fits on eight H100s, then shows which single change to the model breaks it and what replaces it.

A matrix-vector product has one job: keep enough bytes in flight. The answer counts how many that is per SM, then finds the three places a decode GEMV quietly drops below it.

Every dimension of a GEMM tile is pinned by a different piece of hardware. The answer derives the reuse that sets the M and N sides, the swizzle row that sets K, and the register and shared-memory budgets that stop the tile growing.

Eight tiles too many turns a one-wave GEMM into two. The answer prices the waste in microseconds for a real prefill shape, then compares the three fixes, including the one that lives in the scheduler, not the kernel.

Every kernel in the step can run near peak bandwidth while the step itself runs at two thirds of it. The answer reads the three rows of the timeline that tell you why, and turns the gaps into milliseconds.

Both changes move work off the threads: one takes over address generation, the other takes over operand staging. The answer counts the instructions each one removes and shows why a Hopper kernel is organized around barriers instead of thread blocks that do everything.

Hopper's biggest GEMM tiles spend half the register file on the accumulator and feed every operand byte through one SM's shared memory. Blackwell attacks both limits, and the answer counts the bytes each change gives back.

CUDA lets you reserve part of L2 for data you choose, and the KV cache is re-read every step. The answer measures what fits against what a decode step streams, finds the one case where pinning pays, and the kernel change that pays more.

Perplexity averages over the tokens a model finds easy and scores it on text it did not write. The answer builds the evaluation that catches what quantization actually breaks, and shows why a one-point drop on a benchmark can be noise or a real regression.

Reduction order explains the last-bit drift, but a quantized server has a second, much larger leak: a scale computed over the whole batch lets one user's activations change another user's logits. The answer separates the two and fixes both.

Even a format with the same name differs between the two generations: block-scaled FP8 uses FP32 scales on Hopper and power-of-two scales on Blackwell. The answer designs formats, fallbacks and gates so a user routed to either pool gets a model the team has actually tested.

Three parts, and most candidates finish one. The algebra fits on a line, the proof fits in three, and the part interviewers remember is the batch size at which the whole trick runs out of spare compute.

Three knobs, one distribution, and an ordering question that most candidates have never thought about. The answer works one six-token example through every knob and shows the order changing which tokens can be sampled.

At temperature zero, verification is a prefix match, and the whole implementation fits in a dozen vectorized lines. Interviewers grade the three details around it: the bonus token, the KV slots to roll back, and why the output can still differ from plain greedy.

A one-layer drafter that beats an 8B model on acceptance sounds wrong until you ask what it is reading. The answer prices the EAGLE head on Llama 3.3 70B, explains the feature trick, and names the three changes in EAGLE-3.

A drafter that costs nothing and needs no training sounds too good, and on open chat it does nothing. The answer traces a code-rename request step by step, counts accepted tokens under three match-length settings, and says when to turn it off.

Compute acceptance from counter snapshots, reject reset-contaminated windows, and handle EOS before inferring request-step counts. Then decide what a matched-load baseline must measure.

The model never sees the grammar; the sampler does, as a 16 KB bitmask per request per step. The answer builds that mask from an automaton, prices the naive version at ten milliseconds, and shows how precompilation removes almost all of it.

A per-request schema compile stalls the scheduler, a per-step mask must beat the forward pass, and skipping forced characters can push the model off its own tokenization. The answer handles all three with numbers from one realistic schema.

Lossless is a statement about one distribution, and quantization changes which one without raising any error. The answer separates the draft case from the target case, then shows two ways an FP8 deployment breaks exactness without touching the rejection rule.

Code wants eight draft tokens and chat wants two, and a fixed gamma serves both badly. The answer simulates three policies on a mixed workload, finds the gain is 5% with a cheap drafter and 19% with an expensive one, then explains why batching takes most of it away.

The same drafter that triples speed at 3 a.m. costs a third of peak throughput at 6 p.m. if left alone. The answer builds a step-time model for Llama 3.3 70B on eight H100s, derives a draft length per batch bucket, and shows why context length moves the whole table.

Tensor parallelism divides the weight bytes but not the all-reduce count, so each doubling buys less than the one before. The answer prices that fixed cost, then finds the workload where TP=8 wins anyway.

An 8k-token prompt on a 70B model leaves 2.7 GB of KV that has to cross the network before decode can start. The answer shows that the sustained bandwidth is modest, that the latency is the real problem, and which side of the link runs out first.

Tensor parallelism across nodes would all-reduce every activation over the network. Splitting the sequence instead sends only the KV, which for a GQA model is small. The answer sizes that traffic, finds when it hides behind attention, and fixes the causal-mask imbalance that naive splitting creates.

Pulling every replica's copy from object storage turns a 500 GB model into 62.5 TB of reads. The answer finds the three bounds that actually limit a fleet load, shows a pipelined broadcast reaching the tightest one, and lists the startup work that hides behind the bytes.

A 1 TB FP8 model fits on one B200 node and needs four H100 nodes. That single fact sorts the two generations into roles. The answer assigns them, sizes both pools, and then deals with the precision mismatch that shows up when the two halves disagree about the weights.

Decode each token on its own and an emoji streams as two replacement characters. The answer builds the offset-window decoder that fixes it, runs it, and adds the stop-string holdback that most streaming bugs come from.

The mask is a 16 KB bitmap per request. Whether it costs nothing or stalls every step depends on which thread computes it and what it is allowed to overlap with.

About a hundred lines of Python that admit, chunk, preempt, sample and stream. The interviewer is watching where the KV accounting lives and what happens when the pool runs dry.

Short prompts, enormous replies, and a KV footprint that grows every step after admission. The answer sizes the batch from the pool, prices a preemption, and shows why per-user speed, not throughput, is the number to defend.

Most designs draw the router first. The answer that passes does the arithmetic first: tokens per second in each phase, replica-seconds per second, a utilization ceiling, then dollars, and finds that the prompt, not the reply, is two thirds of the bill.

Four boxes are easy to draw. The answer that passes sizes them first, then says exactly which clock starts where, because a TTFT measured inside the engine and a TTFT measured by the client can disagree by a factor of five.

Sending every request with the same prefix to the same replica halves the fleet, until one customer's prompt gets popular. The answer derives the spill threshold from the prefill a cache hit saves, in milliseconds.

Requests-per-minute limits let one tenant with 128K-token prompts take a fifth of a replica's memory per request. The answer prices quotas in the resource tokens actually consume, then isolates the three things tenants share: compute, KV memory and the prefix cache.

A chat fleet sized for its peak sits more than half idle over a day, and a batch API can soak it up. The answer measures the idle replica-hours, finds what limits batch throughput (it is not compute), and decides where the two workloads meet.

Autoscaling cannot save a failover when a replica takes six minutes to come up, so the spare capacity has to exist before the region dies. The answer solves for the smallest fleet that survives any single region loss, and places the spare next to the region most likely to need it.

Input and output tokens burn different resources, so they get different prices. The answer prices each phase from first principles, then shows which assumption moves the list price most.

A canary is only as good as the number of requests behind each verdict. The answer sizes the canary from the statistics of the regression it must catch, then stages the rollout around that number.

A cold 32K prefill on a 70B model barely fits a second of compute on eight H100s, before any queueing. The design that passes makes almost every request warm, then sizes two pools from what remains.

Counting GPUs across two generations gives the wrong plan, and routing by GPU count breaks the SLO on the slower pool. The answer defines one capacity unit at the SLO, then plans, routes and buys in it.

A cheaper GPU and a faster decoder do not automatically produce a cheaper service. Audit a cost ledger, reject misleading wins, and choose a rollout only after quality and latency gates pass.

Inspect tokenized prompts and cache-counter deltas to diagnose a TTFT regression. A runnable prefix-boundary check and a routing follow-up test whether the evidence actually supports the diagnosis.

Greedy decoding is deterministic on paper and the customer has two different answers in hand. The answer finds where the bits change, shows a measured example of it, and turns the finding into a promise the platform can keep.

Perplexity matches the BF16 baseline to two decimals and the needle-in-a-haystack score collapses past 16K. The answer explains why the two checks measure different things, isolates weights, KV and config with a small grid, and shows a measured case of a quantization scheme that only breaks at length.

A regular period is a clue most candidates throw away. The answer turns it into a ranked list of periodic sources, shows why a few slow steps out of hundreds own the p99, and names the one correlation that separates the top two.

The speedup formula has two inputs that vary per request and per moment, and the tail lives where both are bad. The answer computes when speculation turns into a slowdown, shows which requests own the p99, and picks a policy instead of an on/off switch.

The KV pool is preallocated, so the growth is somewhere the engine does not budget. The answer splits memory into what is fixed and what can grow, turns a growth rate into bytes per request, and uses that one number to rule out half the suspects.

A dashboard of forty graphs is a dashboard nobody reads at 2 a.m. The answer builds eight panels in the order an incident unfolds, ties each to the action it decides, and shows how two of them check each other with Little's law.

Unused logical KV pages can coexist with exhausted device memory. Reconcile expanded image tokens, retained embeddings and phase-specific peaks before changing a decoder token limit.

An event can contain metadata, one token or several tokens. Audit a buffered stream without inventing token timestamps, and separate client launch delay from the engine's execution time.

Forty lines of Python, and graders run property tests against them. The answer gets the three invariants right first, then the one copy-on-write case most implementations get backwards.

The attention math is ten lines. The grade comes from where the new keys land, which query head reads which KV head, and the causal mask when a chunk of queries sits at the end of a longer cache.

Everyone can sort and cumsum. The grader feeds you tied logits, p = 1 on a 128K vocabulary and a temperature of zero, and checks whether your kept set depends on the order of the sort.

A scheduler is a loop over three queues and a budget. The answer gets the order of those loops right, shows what the budget costs in milliseconds of TPOT, and tests the invariants a grader will check.

The accept rule is one line. The residual distribution, the bonus token, rows that drafted fewer tokens and a test that can tell a lossless verifier from a subtly biased one are the rest of the grade.

Matching a prefix is a trie walk. The grade is in the edge split, which cached tokens a running request pins, and an eviction loop that turns parents into leaves without ever freeing a slot twice.

A request costs anything from ten tokens to a hundred thousand, and you only learn the real cost when it finishes. The answer reserves at admission, refunds at completion, fails fast on requests that can never fit, and tests the bound the bucket is supposed to guarantee.

Two prompts with the same 16 tokens in block 3 must not share block 3 unless blocks 0 to 2 match too. The answer chains the hashes, keeps freed blocks hittable until memory is needed, evicts tails before heads, and proves a colliding hash cannot hand a request the wrong KV.

The stop string arrives as three tokens, the euro sign arrives as two, and the client has already been sent half of both. The answer holds back exactly the text that could still become a stop, decodes bytes incrementally, and tests the stream against a reference that sees the whole output at once.

At batch 1 a decode attention kernel gives each KV head one thread block and leaves most of the GPU idle. The answer splits the sequence, carries a log-sum-exp with every partial output, merges them exactly, and survives the split that has no keys at all.

A runnable allocator hides an exception-safety defect and a payload-aliasing defect. Reduce each failure to a short operation sequence, then test ownership and content against separate oracles.

A disconnect arrives after launch, then the request ID is reused. Implement deferred reclamation and prove that late or duplicate completions cannot emit stale output or free another request's pages.

The tensor has the right shape and every token ID is valid, yet one request receives another request's continuation. Reproduce the routing error and repair the launch-to-completion contract.

Accepted, emitted and computed tokens are different counts. Trace rejection, full acceptance, EOS and an output cap to find the state that the next forward pass is allowed to read.

One slow connection can stall unrelated generations or retain an unbounded output queue. Implement a byte budget that includes the pending write, then connect overflow and disconnects to engine cancellation.

One division tells you whether a headline number is physics or marketing. The answer runs it, then works out which unstated condition would make the claim true and what to ask the vendor for.

The datasheet ratios are close to each other, so the lazy answer is one speedup for everything. The answer splits it into three regimes, and the biggest win turns out to come from capacity, not bandwidth.

Active parameters describe one token. A decode step serves a batch, and the batch decides how many experts get pulled through HBM. The answer computes that number and shows where the MoE cost advantage actually comes from.

MBU is one division, and that is why it is so easy to get wrong. The answer defines both operands precisely on a real server and then shows six ways a dashboard can report a number 30 points off.

Step time is a straight line in batch size, so throughput is a hyperbola with a ceiling. The answer finds the knee in closed form and shows why the ceiling usually comes from the KV cache, not the tensor cores.

Joules per token is power over throughput, and the power term is the one candidates obsess over. The answer shows why the denominator moves the result by about 50x while the numerator is pinned under a datasheet cap.

Weights and KV are the two numbers everyone knows. The answer that gets the offer accounts for the other few gigabytes, says which ones vLLM measures and which it guesses, and knows where an OOM comes from when the sum looks fine.

Doubling the GPUs past the KV head count halves the weights per GPU and leaves the KV per GPU exactly where it was. The answer derives that, prices the duplicate copy, and picks the parallelism that does not pay it.