Together AI Inference Engineer interview questions

Decode reads every weight once per token. The answer turns that sentence into a millisecond floor, then accounts for the three things that keep a real server above it.

One division tells you whether a headline number is physics or marketing. The answer runs it, then works out which unstated condition would make the claim true and what to ask the vendor for.

The datasheet ratios are close to each other, so the lazy answer is one speedup for everything. The answer splits it into three regimes, and the biggest win turns out to come from capacity, not bandwidth.

Active parameters describe one token. A decode step serves a batch, and the batch decides how many experts get pulled through HBM. The answer computes that number and shows where the MoE cost advantage actually comes from.

MBU is one division, and that is why it is so easy to get wrong. The answer defines both operands precisely on a real server and then shows six ways a dashboard can report a number 30 points off.

The crossover falls out of one division, and the number it gives lines up exactly with something else on the GPU. Spotting that coincidence is what the interviewer is waiting for.

Step time is a straight line in batch size, so throughput is a hyperbola with a ceiling. The answer finds the knee in closed form and shows why the ceiling usually comes from the KV cache, not the tensor cores.

Joules per token is power over throughput, and the power term is the one candidates obsess over. The answer shows why the denominator moves the result by about 50x while the numerator is pinned under a datasheet cap.

Twenty lines of roofline arithmetic get within striking distance of a real server, and the interviewer knows it. What they are scoring is the list of places the model breaks, and how you would catch each one before a capacity plan rests on it.

The formula takes one line. The answer interviewers score is the memory budget around it: what the engine reserves first, what tensor parallelism does to the heads, and why the count you get is a ceiling.

The bound is a one-line argument about where a sequence can be partly full. The block size is a three-way trade between that waste, kernel efficiency and how finely the prefix cache can match.

Weights and KV are the two numbers everyone knows. The answer that gets the offer accounts for the other few gigabytes, says which ones vLLM measures and which it guesses, and knows where an OOM comes from when the sum looks fine.

Doubling the GPUs past the KV head count halves the weights per GPU and leaves the KV per GPU exactly where it was. The answer derives that, prices the duplicate copy, and picks the parallelism that does not pay it.

A prefix cache only hits from the first token forward, so evicting the wrong block can strand gigabytes that stay resident and can never be hit. The answer derives how much, and which eviction order guarantees it never happens.

Absorption is one line of matrix associativity. The part interviewers score is the bill: attention gets more expensive per cached token, and the kernel lands in a roofline regime GQA never visits.

Count the blocks with and without sharing and the saving looks large. The answer that scores also knows when the copy fires, what a misaligned prompt does, and the workload where the saving all but disappears.

All three policies keep the server alive. They differ in who waits, who redoes work, and which percentile absorbs the damage, and the numbers for a 6K-token victim make the choice for you.

The transfer looks like gigabytes per request until you divide by prefill time. Then the prompt length cancels, and the real question becomes which models and which GPUs break the link.

The per-token formula gives a scary number for GQA and a reassuring one for MLA. Put both on real GPUs with tensor parallelism and the reassuring one is the request that does not fit.

Keys and values take the same bytes, yet their errors reach the output by different roads. Trace both roads and the scale layout for each tensor follows, along with the bookkeeping that makes per-channel keys awkward to serve.

A fixed recurrent state still costs memory for every live request and saved checkpoint. Work through a state inventory, then find a restart boundary that every layer can actually restore.

Doubling the batch raised tokens per second by a third and sent goodput to zero. The answer shows why every sequence in a batch shares one step time, and how to find the batch size that maximizes the number users feel.

One 32K-token prompt in front of ten short ones costs each of them more than a second of TTFT. The answer separates the half of the job length you know from the half you do not, and schedules each differently.

Request-fair scheduling hands 96% of the GPU to the tenant with long documents. The answer derives the token weights from the roofline, shows the counter that enforces them per step, and the one rule that stops a tenant from banking credit.

The two targets are coupled: the TPOT target caps the prefill chunk, which caps prefill throughput, which caps the backlog the TTFT target can absorb. The answer builds the step model, derives the admission rule and shows what to do with requests that fail it.

Each draft token is a real token in the step's budget, and verification turns a bandwidth-bound step compute-bound sooner. The answer shows the accounting, the rollback that makes rejection cheap, and the batch size where speculation starts losing.

FlashAttention does the same arithmetic as naive attention and is several times faster. The answer is a three-line recurrence and one IO count, and interviewers want both.

With T new queries over S cached keys, the obvious lower-triangular mask is wrong and still runs without an error. The answer derives the right one from absolute positions and names the library default that gets it wrong.

The one-query recurrence is a warm-up. A real kernel carries a vector of maxima and a vector of sums for a tile of rows, and the rescale becomes a diagonal matrix. The answer writes it, runs it, and shows the edge case that turns it into NaN.

Batch-1 decode at long context gives the GPU eight thread blocks and 132 SMs. The answer counts the waves, picks the split count from a wave-efficiency rule rather than a guess, and derives the merge that makes the result exact.

GQA shrinks the cache by the group size, but a kernel that loops over query heads reads it back group-size times anyway. The answer packs the group into the matrix dimension and shows what that does to intensity and tensor-core utilization.

FlashAttention-2 reaches well under half of an H100's BF16 peak. The answer explains why in one ratio, exponentials against matmuls, and maps each FA3 technique to the stall it hides.

Three quantization points sit inside FP8 attention, and they fail for different reasons: outlier channels in keys, a probability matrix that lives near zero, and accumulators. The answer simulates each one and says which fix addresses which.

Fusing RoPE saves a read and a write of Q and K, which is real but small in prefill. The precision bug is not small: one tensor in the wrong dtype scrambles positional information by a few hundred tokens. The answer sizes both.

Packing removes padding but puts different requests' tokens side by side in one tensor, so the kernel's tile boundaries no longer match sequence boundaries. The answer shows the indexing and a bug that a causal test can never catch.

Review a shape-by-shape timing fixture, catch a ragged-cache correctness failure, and calculate whether selective dispatch pays for itself. Includes a small reference you can run without a GPU.

A matrix-vector product has one job: keep enough bytes in flight. The answer counts how many that is per SM, then finds the three places a decode GEMV quietly drops below it.

Two GEMV kernels do the same arithmetic on the same matrix and differ only in which thread reads which element. One streams at HBM speed; the other turns every warp load into 32 separate transactions.

Every dimension of a GEMM tile is pinned by a different piece of hardware. The answer derives the reuse that sets the M and N sides, the swizzle row that sets K, and the register and shared-memory budgets that stop the tile growing.

RMSNorm does almost no arithmetic, so its cost is bytes, and the eager PyTorch version moves nine times more of them than it needs to. The answer writes the one-pass kernel, tests it on awkward hidden sizes, and says where fusion stops paying.

A Python loop over 256 experts is correct and slow for three separate reasons, only one of which is launch count. The answer builds the expert-sorted layout the fused kernel runs on and counts what it costs in padding.

A 4-bit weight is only cheaper if turning it back into FP16 costs less than the bytes it saved. The answer counts the instruction budget per weight at full HBM bandwidth and shows the bit trick that makes the conversion fit.

Both changes move work off the threads: one takes over address generation, the other takes over operand staging. The answer counts the instructions each one removes and shows why a Hopper kernel is organized around barriers instead of thread blocks that do everything.

Hopper's biggest GEMM tiles spend half the register file on the accumulator and feed every operand byte through one SM's shared memory. Blackwell attacks both limits, and the answer counts the bytes each change gives back.

A kernel time is a distribution measured under conditions, and a reviewer will ask about both. The answer builds the timing loop, says what each line defends against, and adds the one sanity check that catches most bad numbers.

On paper an H200 SXM is an H100 with more and faster memory, so nothing should get slower. The answer works through the four ways it can anyway, from the spec sheet you did not read to the tuning table that does not know the new device's name.

Weight-only INT4 shrinks the bytes but keeps the arithmetic in BF16. The answer finds the batch size where that bargain flips, which does not depend on the model, then explains why a code benchmark notices before perplexity does.

Both formats spend the same eight bits. The answer derives where each one's largest and smallest values come from, then explains why inference almost always picks the narrower of the two.

Finer scales buy accuracy, and the bytes they cost are easy to count. The answer counts them for a 70B model, then shows the cost that matters more: which scales can wait for the epilogue and which ones invade the inner loop.

The two methods ship identical file formats and run on identical kernels, so the difference is entirely in how the rounding was chosen. The answer shows the one design choice that makes one of them fragile when traffic stops looking like the calibration set.

Both formats store the same eight 4-bit values per element. The difference is all in the scale, and the answer shows which of the two scale choices buys more accuracy and what it costs in bits.

A few matrices cause most of the damage when a model is quantized, and keeping them wide costs little memory. The answer builds the sweep that finds them, prices it in GPU hours, and explains why the sum of single-layer damages is not the damage of the mix.

Fine-grained scales let E4M3 carry every tensor, but they change along the reduction axis, so the tensor cores cannot apply them. The answer walks the main loop where they do get applied, prices that work against the MMA, and explains what Blackwell moves into hardware.

Reduction order explains the last-bit drift, but a quantized server has a second, much larger leak: a scale computed over the whole batch lets one user's activations change another user's logits. The answer separates the two and fixes both.

FP8 halves the weights and still leaves almost no memory for the cache. The answer does the budget line by line, picks the format pair that leaves room for real concurrency, and says which hardware change makes the opposite choice correct.

Calibration data reaches each expert only through the router, so the experts that matter for rare domains see almost none of it. The answer counts the tokens, shows the point where a Hessian-based quantizer becomes worse than plain rounding, and builds a recipe that does not depend on luck.

Even a format with the same name differs between the two generations: block-scaled FP8 uses FP32 scales on Hopper and power-of-two scales on Blackwell. The answer designs formats, fallbacks and gates so a user routed to either pool gets a model the team has actually tested.

Three knobs, one distribution, and an ordering question that most candidates have never thought about. The answer works one six-token example through every knob and shows the order changing which tokens can be sampled.

One option drafts four tokens for six percent of a target step but guesses each position blind; the other conditions every guess and costs nearly half a step. The answer prices both on Llama 3.3 70B and finds the batch size where the winner flips.

A draft tree turns a handful of guesses into many candidate continuations, and a mask lets one pass check them all. The answer builds the mask, counts what it costs, and shows where a ten-node tree stops beating a five-token chain.

A one-layer drafter that beats an 8B model on acceptance sounds wrong until you ask what it is reading. The answer prices the EAGLE head on Llama 3.3 70B, explains the feature trick, and names the three changes in EAGLE-3.

DeepSeek V3 ships a drafter it trained alongside the model for free. The answer walks the MTP module's wiring and loss, prices a draft step against a 37.6B-active target, and reproduces the reported 1.8x from first principles.

A drafter that costs nothing and needs no training sounds too good, and on open chat it does nothing. The answer traces a code-rename request step by step, counts accepted tokens under three match-length settings, and says when to turn it off.

The model never sees the grammar; the sampler does, as a 16 KB bitmask per request per step. The answer builds that mask from an automaton, prices the naive version at ten milliseconds, and shows how precompilation removes almost all of it.

Lossless is a statement about one distribution, and quantization changes which one without raising any error. The answer separates the draft case from the target case, then shows two ways an FP8 deployment breaks exactness without touching the rejection rule.

Code wants eight draft tokens and chat wants two, and a fixed gamma serves both badly. The answer simulates three policies on a mixed workload, finds the gain is 5% with a cheap drafter and 19% with an expensive one, then explains why batching takes most of it away.

The same drafter that triples speed at 3 a.m. costs a third of peak throughput at 6 p.m. if left alone. The answer builds a step-time model for Llama 3.3 70B on eight H100s, derives a draft length per batch bucket, and shows why context length moves the whole table.

Two matrix multiplies in a row, and the order of the splits decides whether you communicate once or twice per block. The answer places every split for a 70B layer and prices what the remaining all-reduces send per token.

A token still has to pass through every layer in order, so splitting the layers across nodes cannot make one token faster. The answer shows what pipelining buys instead, prices the bubble, and finds the all-reduce latency at which it beats cross-node tensor parallelism.

Every MoE layer sends each token out to its experts and gathers the results back, twice per layer, 58 times per step. The answer turns DeepSeek V3's config into bytes per token, splits them between NVLink and the NIC, and shows what node-aware dispatch saves.

Tensor parallelism splits a GQA cache by KV head. MLA has no KV heads to split, so TP copies the whole cache onto every GPU without any warning. The answer measures that waste on an H200 node and prices the fix.

An 8k-token prompt on a 70B model leaves 2.7 GB of KV that has to cross the network before decode can start. The answer shows that the sustained bandwidth is modest, that the latency is the real problem, and which side of the link runs out first.

A ring all-reduce is bandwidth-optimal and takes fourteen synchronized steps on eight GPUs. At 16 KB per call, the steps are the cost. The answer builds the latency-bandwidth model, finds the crossover size, and checks it against the threshold vLLM ships.

Same eight GPUs, same weights per GPU, same KV capacity. One layout pays 252 all-reduces per token, the other pays for every layer twice in wall-clock time. The answer works both, then finds the precision choice that forces the second.

Expert parallelism runs at the speed of the busiest GPU, and the router decides which GPU that is. The answer finds the batch size at which skew starts to cost compute, measures what placement and replication recover, and shows why a placement fit to yesterday's traffic decays.

Eight GPUs per NVLink domain shaped every serving layout of the Hopper era. Seventy-two moves the boundary that expert parallelism and KV transfer used to cross over the NIC. The answer prices what moves inside the rack, what still does not fit, and the failure mode that comes with it.

Tensor parallelism across nodes would all-reduce every activation over the network. Splitting the sequence instead sends only the KV, which for a GQA model is small. The answer sizes that traffic, finds when it hides behind attention, and fixes the causal-mask imbalance that naive splitting creates.

Inside one batch there is nothing to overlap: every layer waits for its all-to-all. Split the batch in two and one half computes while the other communicates. The answer models the gain, finds the batch size where splitting makes things slower, and checks it against a shipped default.

Pulling every replica's copy from object storage turns a 500 GB model into 62.5 TB of reads. The answer finds the three bounds that actually limit a fleet load, shows a pipelined broadcast reaching the tightest one, and lists the startup work that hides behind the bytes.

Five processes, one scheduler with no prefill phase, and a busy loop that should never wait on Python. The answer maps each stage to the component that owns it in the current source, not the 2023 paper.

A graph freezes shapes, so engines capture dozens of them and round every batch up. The answer prices that rounding in the regime where decode lives and finds the real costs somewhere else.

A block's identity is a hash of its tokens and of every block before it. The answer traces one request through hashing, lookup, the last-token rule and eviction, with a runnable model of the lookup.

Both reuse KV for shared prefixes, and on a 2K-token chat prompt their hit rates differ by under one percent. The answer finds the differences that do matter, from eviction policy to whether a cache key means anything outside one process.

The GPU finishes a decode step and waits while Python appends tokens, checks stop strings and plans the next batch. The answer shows how the loop reorders that work, where the placeholder tokens live, and the request that pays for it.

Most answers still describe building a TensorRT engine per GPU and batch shape. The current release accepts only the PyTorch backend, and its default admission policy can halve your concurrency. The answer covers both, then says when it is still the right engine.

Decode each token on its own and an emoji streams as two replacement characters. The answer builds the offset-window decoder that fixes it, runs it, and adds the stop-string holdback that most streaming bugs come from.

Eight tenants, eight adapters, one base model and one batch. The answer explains why the extra cost scales with distinct adapters rather than requests, and which default serializes them.

The mask is a 16 KB bitmap per request. Whether it costs nothing or stalls every step depends on which thread computes it and what it is allowed to overlap with.

Same command line, new version, lower throughput. The answer is a method: capture the resolved configuration on both sides, diff the handful of defaults that move every quarter, then pin and bisect.

About a hundred lines of Python that admit, chunk, preempt, sample and stream. The interviewer is watching where the KV accounting lives and what happens when the pool runs dry.

Compare three capture policies using a memory ledger and a matched workload. Recover the missing KV pages, then defend a policy against both capacity and latency constraints.

Most designs draw the router first. The answer that passes does the arithmetic first: tokens per second in each phase, replica-seconds per second, a utilization ceiling, then dollars, and finds that the prompt, not the reply, is two thirds of the bill.

Four boxes are easy to draw. The answer that passes sizes them first, then says exactly which clock starts where, because a TTFT measured inside the engine and a TTFT measured by the client can disagree by a factor of five.

Sending every request with the same prefix to the same replica halves the fleet, until one customer's prompt gets popular. The answer derives the spill threshold from the prefill a cache hit saves, in milliseconds.

A web-service autoscaler watching GPU utilization will scale this fleet too late and in the wrong direction. The answer turns the four-minute load time into a headroom number, then decides what absorbs the bursts no scaler can catch.

Requests-per-minute limits let one tenant with 128K-token prompts take a fifth of a replica's memory per request. The answer prices quotas in the resource tokens actually consume, then isolates the three things tenants share: compute, KV memory and the prefix cache.

A chat fleet sized for its peak sits more than half idle over a day, and a batch API can soak it up. The answer measures the idle replica-hours, finds what limits batch throughput (it is not compute), and decides where the two workloads meet.

Five hundred merged fine-tunes need five hundred GPUs before the first request. Served as adapters on a shared base they need a few dozen. The answer sizes the adapters, places them in a three-tier memory hierarchy, and finds the one setting that silently multiplies their memory by eight.

Twenty models with two replicas each is a fleet that is mostly idle. The answer measures how idle, then tiers the models by load so the long tail shares GPUs without hurting the models that pay the bills.

Input and output tokens burn different resources, so they get different prices. The answer prices each phase from first principles, then shows which assumption moves the list price most.

A cold 32K prefill on a 70B model barely fits a second of compute on eight H100s, before any queueing. The design that passes makes almost every request warm, then sizes two pools from what remains.

Counting GPUs across two generations gives the wrong plan, and routing by GPU count breaks the SLO on the slower pool. The answer defines one capacity unit at the SLO, then plans, routes and buys in it.

A cheaper GPU and a faster decoder do not automatically produce a cheaper service. Audit a cost ledger, reject misleading wins, and choose a rollout only after quality and latency gates pass.

Users see the first token quickly and then the stream stutters. The answer builds the decode step from bytes, shows which cause produces a bimodal latency histogram, and names the one metric that tells the top two apart.

Prompts fit, but their outputs exhaust the pool. Work through a page-level memory ledger, choose the next diagnostic check, and test a concurrency fix against queueing and goodput.

FP8 halves the weight bytes, and the dashboard says the replica got slower. The answer computes what FP8 should have bought, splits the gap into kernel, scale and config terms with one profiler trace, and catches the regression that is not in the GPU at all.

Greedy decoding is deterministic on paper and the customer has two different answers in hand. The answer finds where the bits change, shows a measured example of it, and turns the finding into a promise the platform can keep.

Same model, same image, same instance type, and one replica drags the fleet's p99. The answer first proves the replica is slower at equal load rather than busier, then walks the hardware causes from GPU clocks to a link that trained at the wrong speed.

Both numbers can be true at once. The answer shows how to turn the gap into a short list of measurable dilutions, which one usually eats most of the 2x, and the replay test that settles it.

A hang gives you no error, no log line and no metric. The answer shows how to make every rank say where it is stuck, why a wrong network transport makes things slow but rarely hangs them, and the order that finds the cause in minutes.

The speedup formula has two inputs that vary per request and per moment, and the tail lives where both are bad. The answer computes when speculation turns into a slowdown, shows which requests own the p99, and picks a policy instead of an on/off switch.

Small models make the GPU step so short that host work stops hiding behind it. The answer shows how to prove that with a timeline instead of a hunch, how much throughput a few milliseconds of Python cost, and which fixes help in which order.

A dashboard of forty graphs is a dashboard nobody reads at 2 a.m. The answer builds eight panels in the order an incident unfolds, ties each to the action it decides, and shows how two of them check each other with Little's law.

Use cold-schema and warm-batch traces to separate compilation from recurring stalls. Identify the exposed delay, choose a targeted experiment, and preserve output correctness.

Compare expert counts with rank-level completion times. Explain why the longest combine span can occur on a lightly loaded rank, then test a placement change against a communication control.

Reconstruct token arrival from a request-correlated trace. Separate transfer overlap from decode admission, and show how first-token delivery changes the metric that exposes a handoff stall.

Replay the same burst through affinity, round-robin and predicted-finish routing. Then remove local reservations or delay an eviction notice and explain exactly why the decisions change.

An event can contain metadata, one token or several tokens. Audit a buffered stream without inventing token timestamps, and separate client launch delay from the engine's execution time.

Everyone can sort and cumsum. The grader feeds you tied logits, p = 1 on a 128K vocabulary and a temperature of zero, and checks whether your kept set depends on the order of the sort.

The accept rule is one line. The residual distribution, the bonus token, rows that drafted fewer tokens and a test that can tell a lossless verifier from a subtly biased one are the rest of the grade.

RoPE is a rotation per pair of dimensions. The bugs live in which dimensions pair up, which position each row of a ragged batch is at, what left padding does to position ids, and the precision of the angle at position 100,000.

Two nibbles per byte and a scale per 128 weights sounds like bookkeeping. The grader checks the sign extension, the nibble order, what happens when K is odd or not a multiple of the group, and whether your error test has a bound or just a vibe.

The tutorial kernel loads a whole row into one block and breaks the moment the row is a 128,256-entry vocabulary. The answer writes the looped version, keeps it free of NaNs when a whole tile is masked, and tests it the way a grader will.

A request costs anything from ten tokens to a hundred thousand, and you only learn the real cost when it finishes. The answer reserves at admission, refunds at completion, fails fast on requests that can never fit, and tests the bound the bucket is supposed to guarantee.

At batch 1 a decode attention kernel gives each KV head one thread block and leaves most of the GPU idle. The answer splits the sequence, carries a log-sum-exp with every partial output, merges them exactly, and survives the split that has no keys at all.

The tensor has the right shape and every token ID is valid, yet one request receives another request's continuation. Reproduce the routing error and repair the launch-to-completion contract.

A fixed global seed reproduces a launch order, not a request. Build a random-draw address that survives compaction, extra draft work and continuation, then state what it cannot guarantee.

The textbook IO count says FlashAttention re-reads K and V once per query block, which at 8K is only 2x better than naive. The answer explains why the real HBM number is 65x lower, and which cache makes the difference.

Storing the attention matrix for backward costs gigabytes per layer; recomputing it costs extra FLOPs. The answer derives the gradient, shows the one scalar per row that makes recomputation exact, and the identity that removes the last N-squared term.

The indirection adds 4 bytes per 64 KiB of KV, so the byte count barely moves. The answer shows where the real cost hides: dependent loads, tiles that span several pages, and copy engines that want one contiguous box.

A 4K window over 32K tokens keeps one eighth of the keys, and the speedup over causal attention is not 8x. The answer counts tiles exactly, classifies which need masking, and explains where the rest of the gap goes.

An error that grows with context length is a running sum losing increments, and BF16 stops adding 1 to a total at 256. The answer turns the symptom into a bisection, finds the state held in the wrong dtype, and shows the error curve that proves it.

At 1M tokens a 70B GQA model needs 320 GiB of KV per request and spends over 90% of prefill FLOPs in attention. The answer lays out a plan that fits on eight H100s, then shows which single change to the model breaks it and what replaces it.

Occupancy is the minimum of four separate budgets, and candidates usually compute one of them. The answer works all four for an H100, finds which one binds, and then asks whether raising occupancy would help at all.

The one-column pad is the textbook fix, and it works for FP32 scalars. The answer derives why, then shows what happens with BF16 and with 16-byte vector stores, where the pad stops working and kernels swizzle instead.