latency
Inference interview questions tagged latency, across every topic.
5 questions · 0 unlocked for you
Concepts behind "latency"
Concept pages for the mechanisms these questions lean on.
Foundational
Latency Decomposition: Queue, Prefill, DecodeHow to break LLM request latency into its parts: TTFT is network plus queue plus prefill, end-to-end adds output length times inter-token latency. Which vLLM metrics measure each part, why percentiles do not add, and how Little's law cross-checks a dashboard.🩺 Performance Debugging
Advanced
CUDA Graphs and Launch Overhead in DecodeA decode step is hundreds of small kernels, and once tensor parallelism or a small model shrinks the GPU time per step, the CPU cost of launching them becomes the step time. CUDA graphs record the whole step once and replay it with one launch, which is why every serving engine captures decode graphs per batch-size bucket.🟩 GPU Performance🔒 Premium
Advanced
Inference Benchmarks That Do Not LieAnalyze scheduled-arrival and send timestamps, account for failures, and reproduce goodput from a small request trace. Then use an executable queue model to see what a closed-loop client hides before designing a matched-load benchmark.🩺 Performance Debugging🔒 Premium