xAI (SpaceXSI) Inference Engineer interview questions

The datasheet ratios are close to each other, so the lazy answer is one speedup for everything. The answer splits it into three regimes, and the biggest win turns out to come from capacity, not bandwidth.

Active parameters describe one token. A decode step serves a batch, and the batch decides how many experts get pulled through HBM. The answer computes that number and shows where the MoE cost advantage actually comes from.

At 1M tokens a 70B GQA model needs 320 GiB of KV per request and spends over 90% of prefill FLOPs in attention. The answer lays out a plan that fits on eight H100s, then shows which single change to the model breaks it and what replaces it.

Every MoE layer sends each token out to its experts and gathers the results back, twice per layer, 58 times per step. The answer turns DeepSeek V3's config into bytes per token, splits them between NVLink and the NIC, and shows what node-aware dispatch saves.

Expert parallelism runs at the speed of the busiest GPU, and the router decides which GPU that is. The answer finds the batch size at which skew starts to cost compute, measures what placement and replication recover, and shows why a placement fit to yesterday's traffic decays.

Eight GPUs per NVLink domain shaped every serving layout of the Hopper era. Seventy-two moves the boundary that expert parallelism and KV transfer used to cross over the NIC. The answer prices what moves inside the rack, what still does not fit, and the failure mode that comes with it.

A 1 TB FP8 model fits on one B200 node and needs four H100 nodes. That single fact sorts the two generations into roles. The answer assigns them, sizes both pools, and then deals with the precision mismatch that shows up when the two halves disagree about the weights.

Both reuse KV for shared prefixes, and on a 2K-token chat prompt their hit rates differ by under one percent. The answer finds the differences that do matter, from eviction policy to whether a cache key means anything outside one process.

The GPU finishes a decode step and waits while Python appends tokens, checks stop strings and plans the next batch. The answer shows how the loop reorders that work, where the placeholder tokens live, and the request that pays for it.

Short prompts, enormous replies, and a KV footprint that grows every step after admission. The answer sizes the batch from the pool, prices a preemption, and shows why per-user speed, not throughput, is the number to defend.

The formula takes one line. The answer interviewers score is the memory budget around it: what the engine reserves first, what tensor parallelism does to the heads, and why the count you get is a ceiling.

The bound is a one-line argument about where a sequence can be partly full. The block size is a three-way trade between that waste, kernel efficiency and how finely the prefix cache can match.

Weights and KV are the two numbers everyone knows. The answer that gets the offer accounts for the other few gigabytes, says which ones vLLM measures and which it guesses, and knows where an OOM comes from when the sum looks fine.

FP8 halves the cache and the attention bytes, and on short prompts it looks free. The answer explains why the scale matters less than people think for rounding and more than they think for clipping, and why the failures hide at 128K.

Doubling the GPUs past the KV head count halves the weights per GPU and leaves the KV per GPU exactly where it was. The answer derives that, prices the duplicate copy, and picks the parallelism that does not pay it.

Half the layers capped at a 128-token window sounds like a big saving at 128K. The arithmetic says the saving is set by the layer ratio, not the window, and the allocator decides whether you get it at all.

Recomputing a prefix costs FLOPs proportional to the model, fetching it costs bytes proportional to the cache. One ratio decides it, and the answer shows by how much fetching wins and the three situations where it still loses.

A prefix cache only hits from the first token forward, so evicting the wrong block can strand gigabytes that stay resident and can never be hit. The answer derives how much, and which eviction order guarantees it never happens.

Absorption is one line of matrix associativity. The part interviewers score is the bill: attention gets more expensive per cached token, and the kernel lands in a roofline regime GQA never visits.

Count the blocks with and without sharing and the saving looks large. The answer that scores also knows when the copy fires, what a misaligned prompt does, and the workload where the saving all but disappears.

All three policies keep the server alive. They differ in who waits, who redoes work, and which percentile absorbs the damage, and the numbers for a 6K-token victim make the choice for you.

The transfer looks like gigabytes per request until you divide by prefill time. Then the prompt length cancels, and the real question becomes which models and which GPUs break the link.

The per-token formula gives a scary number for GQA and a reassuring one for MLA. Put both on real GPUs with tensor parallelism and the reassuring one is the request that does not fit.

Keys and values take the same bytes, yet their errors reach the output by different roads. Trace both roads and the scale layout for each tensor follows, along with the bookkeeping that makes per-channel keys awkward to serve.

Each tier is worth exactly the recompute it avoids minus the fetch it costs. Work that out per token and the design writes itself, including the tier that should not exist for small models and the SSD that wears out in a week.

A fixed recurrent state still costs memory for every live request and saved checkpoint. Work through a state inventory, then find a restart boundary that every layer can actually restore.