AMD Inference Engineer interview questions

Every weight GEMM ties, attention sits at a constant the batch cannot move, and the ranking flips at batch 8. The answer then separates what fusion can fix from what only the model architecture or the batch can.

FP8 halves the cache and the attention bytes, and on short prompts it looks free. The answer explains why the scale matters less than people think for rounding and more than they think for clipping, and why the failures hide at 128K.

Keys and values take the same bytes, yet their errors reach the output by different roads. Trace both roads and the scale layout for each tensor follows, along with the bookkeeping that makes per-channel keys awkward to serve.

The one-query recurrence is a warm-up. A real kernel carries a vector of maxima and a vector of sums for a tile of rows, and the rescale becomes a diagonal matrix. The answer writes it, runs it, and shows the edge case that turns it into NaN.

The textbook IO count says FlashAttention re-reads K and V once per query block, which at 8K is only 2x better than naive. The answer explains why the real HBM number is 65x lower, and which cache makes the difference.

Three quantization points sit inside FP8 attention, and they fail for different reasons: outlier channels in keys, a probability matrix that lives near zero, and accumulators. The answer simulates each one and says which fix addresses which.

An error that grows with context length is a running sum losing increments, and BF16 stops adding 1 to a total at 256. The answer turns the symptom into a bisection, finds the state held in the wrong dtype, and shows the error curve that proves it.

Review a shape-by-shape timing fixture, catch a ragged-cache correctness failure, and calculate whether selective dispatch pays for itself. Includes a small reference you can run without a GPU.

Occupancy is the minimum of four separate budgets, and candidates usually compute one of them. The answer works all four for an H100, finds which one binds, and then asks whether raising occupancy would help at all.

The one-column pad is the textbook fix, and it works for FP32 scalars. The answer derives why, then shows what happens with BF16 and with 16-byte vector stores, where the pad stops working and kernels swizzle instead.

Every dimension of a GEMM tile is pinned by a different piece of hardware. The answer derives the reuse that sets the M and N sides, the swizzle row that sets K, and the register and shared-memory budgets that stop the tile growing.

A Python loop over 256 experts is correct and slow for three separate reasons, only one of which is launch count. The answer builds the expert-sorted layout the fused kernel runs on and counts what it costs in padding.

A 4-bit weight is only cheaper if turning it back into FP16 costs less than the bytes it saved. The answer counts the instruction budget per weight at full HBM bandwidth and shows the bit trick that makes the conversion fit.

A kernel time is a distribution measured under conditions, and a reviewer will ask about both. The answer builds the timing loop, says what each line defends against, and adds the one sanity check that catches most bad numbers.

Both formats spend the same eight bits. The answer derives where each one's largest and smallest values come from, then explains why inference almost always picks the narrower of the two.

Finer scales buy accuracy, and the bytes they cost are easy to count. The answer counts them for a 70B model, then shows the cost that matters more: which scales can wait for the epilogue and which ones invade the inner loop.

A handful of channels with huge activations wreck the INT8 grid for every other channel in the token. The answer shows why the obvious fix cannot run on a tensor core, what SmoothQuant does instead, and why FP8 makes most of the problem disappear.

Both formats store the same eight 4-bit values per element. The difference is all in the scale, and the answer shows which of the two scale choices buys more accuracy and what it costs in bits.

Fine-grained scales let E4M3 carry every tensor, but they change along the reduction axis, so the tensor cores cannot apply them. The answer walks the main loop where they do get applied, prices that work against the MMA, and explains what Blackwell moves into hardware.

Lossless is a statement about one distribution, and quantization changes which one without raising any error. The answer separates the draft case from the target case, then shows two ways an FP8 deployment breaks exactness without touching the rejection rule.

A ring all-reduce is bandwidth-optimal and takes fourteen synchronized steps on eight GPUs. At 16 KB per call, the steps are the cost. The answer builds the latency-bandwidth model, finds the crossover size, and checks it against the threshold vLLM ships.

The scheduler did not change; the code that turns its decisions into GPU tensors did. The answer covers where per-request state now lives, why Triton became a hard dependency, and the feature list that silently puts you back on the old runner.

Two nibbles per byte and a scale per 128 weights sounds like bookkeeping. The grader checks the sign extension, the nibble order, what happens when K is odd or not a multiple of the group, and whether your error test has a bound or just a vibe.

The tutorial kernel loads a whole row into one block and breaks the moment the row is a 128,256-entry vocabulary. The answer writes the looped version, keeps it free of NaNs when a whole tile is masked, and tests it the way a grader will.

At batch 1 a decode attention kernel gives each KV head one thread block and leaves most of the GPU idle. The answer splits the sequence, carries a log-sum-exp with every partial output, merges them exactly, and survives the split that has no keys at all.

FlashAttention does the same arithmetic as naive attention and is several times faster. The answer is a three-line recurrence and one IO count, and interviewers want both.

With T new queries over S cached keys, the obvious lower-triangular mask is wrong and still runs without an error. The answer derives the right one from absolute positions and names the library default that gets it wrong.

Storing the attention matrix for backward costs gigabytes per layer; recomputing it costs extra FLOPs. The answer derives the gradient, shows the one scalar per row that makes recomputation exact, and the identity that removes the last N-squared term.

Batch-1 decode at long context gives the GPU eight thread blocks and 132 SMs. The answer counts the waves, picks the split count from a wave-efficiency rule rather than a guess, and derives the merge that makes the result exact.

The indirection adds 4 bytes per 64 KiB of KV, so the byte count barely moves. The answer shows where the real cost hides: dependent loads, tiles that span several pages, and copy engines that want one contiguous box.

GQA shrinks the cache by the group size, but a kernel that loops over query heads reads it back group-size times anyway. The answer packs the group into the matrix dimension and shows what that does to intensity and tensor-core utilization.

FlashAttention-2 reaches well under half of an H100's BF16 peak. The answer explains why in one ratio, exponentials against matmuls, and maps each FA3 technique to the stall it hides.

A 4K window over 32K tokens keeps one eighth of the keys, and the speedup over causal attention is not 8x. The answer counts tiles exactly, classifies which need masking, and explains where the rest of the gap goes.