TL;DR: Temperature rescales logits (it changes the shape and keeps every token), top-k and top-p truncate the support, and the order matters: on a six-token example at , top-p 0.9 keeps four tokens if applied before temperature and five if applied after. vLLM V1 (checked at v0.31.0) applies penalties, then temperature, then min-p, then top-k, then top-p, then samples.
How to approach it
Treat each knob as an operation on one vector of logits and say what it preserves. Temperature is a reshaping; top-k and top-p are masks. Then ask the clarifying question that makes this a real engineering answer: is the goal to explain behavior, or to reproduce another engine's outputs token for token? The second needs the exact order, the tie rule, and what happens at temperature zero.
A strong answer
Start from logits . Temperature divides them before the softmax:
sharpens the distribution, flattens it, and as it collapses to argmax. No token is ever removed, so a 0.1% token stays possible at any finite temperature. Top-k keeps the largest logits and sets the rest to . Top-p (nucleus) keeps the smallest set of highest-probability tokens whose mass reaches , so its width adapts: two tokens on a confident step, hundreds on an uncertain one. Min-p keeps tokens with probability at least min-p times the top token's probability, which scales the cutoff with the model's confidence.
Work one example with logits :
T = 1.0 p = [0.446 0.271 0.164 0.060 0.037 0.022] cum = 0.446 0.717 0.881 0.941
T = 0.7 p = [0.547 0.268 0.131 0.031 0.015 0.008] cum = 0.547 0.815 0.946
T = 1.5 p = [0.355 0.254 0.182 0.094 0.067 0.048] cum = 0.355 0.609 0.791 0.885 0.952
top-p 0.9 kept set: T = 0.7 -> 3 tokens, T = 1.0 -> 4 tokens, T = 1.5 -> 5 tokens
renormalized at T = 0.7: 0.547/0.946 = 0.578, 0.268/0.946 = 0.283, 0.131/0.946 = 0.138
sanity: a 200,000-draw run of the sampler below gave 0.580, 0.281, 0.139
Now the ordering. At , apply top-p first on the untempered distribution: the cumulative mass reaches 0.941 at the fourth token, so four tokens survive and temperature then spreads probability across those four. Apply temperature first: mass reaches 0.952 only at the fifth token, so five survive. The fifth token gets 7% of draws in one order and zero in the other. Temperature and top-k commute (dividing by a positive never changes the ranking), but temperature and top-p do not, because top-p reads probabilities.
| Engine step (vLLM V1, v0.31.0) | What it does |
|---|---|
| allowed-token whitelist, bad words | hard masks |
| min-tokens, logit bias | change the argmax, so before greedy |
| repetition, frequency, presence penalties | edit logits from output history |
| greedy short-circuit | rows with take argmax here |
| temperature | divides logits |
| min-p | first argmax-invariant processor |
| top-k, then top-p | masks on the sorted row |
| sample | argmax of with |
The last row is a detail that interviewers like. vLLM avoids torch.multinomial because it forces a GPU-to-CPU sync; dividing probabilities by exponential noise and taking the argmax draws exactly from with no sync. A reference sampler in the same order:
import numpy as np
def sample(logits, temperature=1.0, top_k=0, top_p=1.0, min_p=0.0, rng=None):
"""One row of logits [V] -> token id, in vLLM V1's order (checked v0.31.0):
temperature, then min_p, then top-k, then top-p, then sample."""
z = logits.astype(np.float64)
if temperature < 1e-5:
return int(np.argmax(z)) # greedy short-circuit
z = z / temperature
p = np.exp(z - z.max()); p /= p.sum()
if min_p > 0:
z = np.where(p < min_p * p.max(), -np.inf, z)
if 0 < top_k < len(z):
kth = np.sort(z)[-top_k]
z = np.where(z < kth, -np.inf, z) # ties at the k-th value survive
if top_p < 1.0:
order = np.argsort(z) # ascending, like vLLM
p = np.exp(z[order] - z.max()); p /= p.sum()
drop = np.cumsum(p) <= 1 - top_p
drop[-1] = False # always keep the top token
z[order[drop]] = -np.inf
p = np.exp(z - z.max()); p /= p.sum()
rng = rng or np.random.default_rng()
q = rng.exponential(size=p.shape) # argmax(p / Exp(1)) ~ p
return int(np.argmax(p / q))
Two edge cases it handles the way the engine does: top-k with a tie at the -th logit keeps every tied token (so top_k=1 on two equal maxima can return either), and a tiny top-p still keeps the single most likely token.
When would the answer reverse? If the task is matching another provider's outputs, do not assume this order. Hugging Face, other engines and hosted APIs have each made their own choices about whether truncation sees tempered or raw probabilities, and a reproduction test should pin both the order and the tie rule before comparing samples.
What interviewers probe next
"Why is temperature 0 handled separately instead of dividing by a tiny number?" Dividing by zero is undefined and dividing by overflows in FP16 or BF16, so engines branch to argmax below an epsilon.
"Top-p over a 128K vocabulary needs a sort. What does that cost at batch 256?" A full sort per row is the slow path, so engines use top-k-first pruning or specialized kernels (FlashInfer's sampling kernels in vLLM) that avoid materializing the full sorted vocabulary.
"Where do repetition penalties go relative to temperature?" Before it, on raw logits, because they are meant to change the greedy choice too.
How it's scored
Weak: says temperature "controls creativity" and treats top-p as a fixed number of tokens.
Hire: writes the tempered softmax, explains that top-p's width adapts to confidence, and shows that temperature and top-k commute.
Strong Hire: constructs the counterexample where top-p and temperature do not commute, names the engine's order with a version, and mentions the tie rule and the sync-free sampling trick.
Common mistakes
- Claiming high temperature can sample tokens that top-p removed. Once a logit is , no temperature brings it back.
- Applying top-p to logits instead of probabilities, which makes the cutoff meaningless.
- Forgetting to renormalize after masking, so reported probabilities sum to 0.946 instead of 1.
- Testing reproducibility with temperature 0 and assuming it covers the sampled path, which never runs the masks.