Advanced

Nondeterminism and Batch Invariance

Why an LLM endpoint gives different outputs at temperature 0: floating-point addition is not associative, kernels pick their reduction strategy (split-K, split reductions, split-KV) from the batch shape, and server load sets the batch. What atomics do and do not explain, what batch-invariant kernels change, and what they cost.

Get full Premium access · ₹2,000 / $25

Every worked answer and derivation in all 12 topics, every concept page, and the full practice-test bank.

6 months · One payment · No auto-renewal

RELATED CONCEPTS

Inference Benchmarks That Do Not Lie Split-KV Decoding (Flash-Decoding)Number Formats for Inference: BF16, FP8, INT4, FP4 Property-Testing Engine Components Latency Decomposition: Queue, Prefill, Decode Sampling: Temperature, Top-k and Top-p

PRACTICE THIS IN REAL QUESTIONS

Attention KernelsDerive online softmax, prove it is exact, and explain why it makes FlashAttention fast without reducing FLOPs.→Performance Debugging & BenchmarkingAfter a prompt-template deploy, p99 TTFT went from about 100 ms to over a second while GPU utilization did not move. Debug it.→Performance Debugging & BenchmarkingTTFT is fine but inter-token latency is bad. Decompose the decode step and rank the causes.→Attention KernelsImplement online softmax for a whole tile of queries, not one query, and write the rescaling step in matrix form.→Performance Debugging & BenchmarkingvLLM fails to start with an out-of-memory error on a model that should fit. Walk through the startup memory profile.→Implement-from-Scratch CodingImplement batched top-p sampling with temperature. It must handle ties at the cutoff, p = 1, temperature 0 and masked tokens.→