Jane Street Inference Engineer interview questions

Every weight GEMM ties, attention sits at a constant the batch cannot move, and the ranking flips at batch 8. The answer then separates what fusion can fix from what only the model architecture or the batch can.

The 70% rule of thumb falls out of one formula once you plug in a prefill time and a p99 target. The answer derives it, shows how prompt-length variance moves it to 50%, and lists the four ways an LLM engine breaks the textbook queue.

An error that grows with context length is a running sum losing increments, and BF16 stops adding 1 to a total at 256. The answer turns the symptom into a bisection, finds the state held in the wrong dtype, and shows the error curve that proves it.

Occupancy is the minimum of four separate budgets, and candidates usually compute one of them. The answer works all four for an H100, finds which one binds, and then asks whether raising occupancy would help at all.

A kernel time is a distribution measured under conditions, and a reviewer will ask about both. The answer builds the timing loop, says what each line defends against, and adds the one sanity check that catches most bad numbers.

About a hundred lines of Python that admit, chunk, preempt, sample and stream. The interviewer is watching where the KV accounting lives and what happens when the pool runs dry.

Decode reads every weight once per token. The answer turns that sentence into a millisecond floor, then accounts for the three things that keep a real server above it.

Two FLOPs per parameter is the number everyone quotes. The answer interviewers want rebuilds it from the config, finds the parameters that do no arithmetic, and names the context length where the rule breaks.

Decode and prefill run the same weights through the same kernels, yet one waits on memory and the other on tensor cores. The answer finds the prompt length where they trade places and turns it into a TTFT.

Batching rescues the weight GEMMs at decode but never the attention kernel, and one head-count ratio explains why. The answer derives it, then shows how MLA moves attention to within a hair of the H100 ridge.

One division tells you whether a headline number is physics or marketing. The answer runs it, then works out which unstated condition would make the claim true and what to ask the vendor for.

The datasheet ratios are close to each other, so the lazy answer is one speedup for everything. The answer splits it into three regimes, and the biggest win turns out to come from capacity, not bandwidth.

Active parameters describe one token. A decode step serves a batch, and the batch decides how many experts get pulled through HBM. The answer computes that number and shows where the MoE cost advantage actually comes from.

MBU is one division, and that is why it is so easy to get wrong. The answer defines both operands precisely on a real server and then shows six ways a dashboard can report a number 30 points off.

A 60% MBU number is a symptom, not a diagnosis. The answer rebuilds the floor from bytes, then splits the missing 40% into buckets a profiler can confirm and a team can own.

The crossover falls out of one division, and the number it gives lines up exactly with something else on the GPU. Spotting that coincidence is what the interviewer is waiting for.

Step time is a straight line in batch size, so throughput is a hyperbola with a ceiling. The answer finds the knee in closed form and shows why the ceiling usually comes from the KV cache, not the tensor cores.

Joules per token is power over throughput, and the power term is the one candidates obsess over. The answer shows why the denominator moves the result by about 50x while the numerator is pinned under a datasheet cap.

A 2.5x gap at batch 1 almost never comes from the GPU being slow. The answer sorts the suspects by what a single profiler trace shows and puts a millisecond figure on each.

Twenty lines of roofline arithmetic get within striking distance of a real server, and the interviewer knows it. What they are scoring is the list of places the model breaks, and how you would catch each one before a capacity plan rests on it.

FlashAttention does the same arithmetic as naive attention and is several times faster. The answer is a three-line recurrence and one IO count, and interviewers want both.

With T new queries over S cached keys, the obvious lower-triangular mask is wrong and still runs without an error. The answer derives the right one from absolute positions and names the library default that gets it wrong.