gpu memory utilization
Inference interview questions tagged gpu memory utilization, across every topic.
2 questions · 2 unlocked for you
Concepts behind "gpu memory utilization"
Concept pages for the mechanisms these questions lean on.
Foundational
The KV Cache: Bytes per Token and the Memory BudgetThe KV cache stores every past token's keys and values so decode never recomputes them. Its size per token is fixed by the attention design, and the memory left after weights and the engine's reserve decides how many sequences fit, which is the number that sets batch size, concurrency and when preemption starts.🧮 KV Cache & Memory
Advanced
Serving Engine Knobs and Their DefaultsA handful of engine settings decide how much KV fits, how long a step can get and how many requests run at once. This page lists the defaults in vLLM v0.31.0 and SGLang v0.5.21 as read from source, shows which knob binds first for Llama 3.3 70B on four H100s, and gives the order in which to tune them.⚙️ Serving Engines🔒 Premium