Skip to content

memory budget

Inference interview questions tagged memory budget, across every topic.

2 questions · 0 unlocked for you
Open full answer →
Open full answer →

Concepts behind "memory budget"

Concept pages for the mechanisms these questions lean on.

Foundational
🧮 KV Cache & Memory
The KV Cache: Bytes per Token and the Memory BudgetThe KV cache stores every past token's keys and values so decode never recomputes them. Its size per token is fixed by the attention design, and the memory left after weights and the engine's reserve decides how many sequences fit, which is the number that sets batch size, concurrency and when preemption starts.
Explore the full curriculum →