Google (TPU / Cloud AI infrastructure) Inference Engineer interview questions

Decode reads every weight once per token. The answer turns that sentence into a millisecond floor, then accounts for the three things that keep a real server above it.

Two FLOPs per parameter is the number everyone quotes. The answer interviewers want rebuilds it from the config, finds the parameters that do no arithmetic, and names the context length where the rule breaks.

Decode and prefill run the same weights through the same kernels, yet one waits on memory and the other on tensor cores. The answer finds the prompt length where they trade places and turns it into a TTFT.

Batching rescues the weight GEMMs at decode but never the attention kernel, and one head-count ratio explains why. The answer derives it, then shows how MLA moves attention to within a hair of the H100 ridge.

One division tells you whether a headline number is physics or marketing. The answer runs it, then works out which unstated condition would make the claim true and what to ask the vendor for.

The datasheet ratios are close to each other, so the lazy answer is one speedup for everything. The answer splits it into three regimes, and the biggest win turns out to come from capacity, not bandwidth.

Active parameters describe one token. A decode step serves a batch, and the batch decides how many experts get pulled through HBM. The answer computes that number and shows where the MoE cost advantage actually comes from.

MBU is one division, and that is why it is so easy to get wrong. The answer defines both operands precisely on a real server and then shows six ways a dashboard can report a number 30 points off.

A 60% MBU number is a symptom, not a diagnosis. The answer rebuilds the floor from bytes, then splits the missing 40% into buckets a profiler can confirm and a team can own.

The crossover falls out of one division, and the number it gives lines up exactly with something else on the GPU. Spotting that coincidence is what the interviewer is waiting for.

Step time is a straight line in batch size, so throughput is a hyperbola with a ceiling. The answer finds the knee in closed form and shows why the ceiling usually comes from the KV cache, not the tensor cores.

Joules per token is power over throughput, and the power term is the one candidates obsess over. The answer shows why the denominator moves the result by about 50x while the numerator is pinned under a datasheet cap.

A 2.5x gap at batch 1 almost never comes from the GPU being slow. The answer sorts the suspects by what a single profiler trace shows and puts a millisecond figure on each.

Every weight GEMM ties, attention sits at a constant the batch cannot move, and the ranking flips at batch 8. The answer then separates what fusion can fix from what only the model architecture or the batch can.

Twenty lines of roofline arithmetic get within striking distance of a real server, and the interviewer knows it. What they are scoring is the list of places the model breaks, and how you would catch each one before a capacity plan rests on it.

A matrix-vector product has one job: keep enough bytes in flight. The answer counts how many that is per SM, then finds the three places a decode GEMV quietly drops below it.