Reserve the page a request has not written yet

TL;DR: Reserve against the declared maximum context for this course's conservative policy. A 2,048-token prompt plus 257 generated tokens needs 145 pages of 16 tokens, even though the prompt alone needs only 128.

Where you are. You have a latency contract. Now make memory admission a promise the worker can keep.

Separate the pool from its owners

The specified decoder has 32 layers, eight KV heads per layer, a head dimension of 128 and BF16 keys and values. These are synthetic architecture inputs. We make no assertion about model weights or an accelerator's total memory. The worker has an already allocated KV pool of 512 MiB after all other reservations.

KV bytes/token = 32 layers × 8 KV heads × 128 × 2 K/V × 2 bytes
               = 131072 bytes = 128 KiB
Page bytes     = 16 tokens × 128 KiB = 2 MiB
Pool pages     = 512 MiB / 2 MiB = 256 pages
Request pages  = ceil((2048 + 257) / 16) = 145 pages

The last page is mostly empty. It still consumes a physical page. Ignoring it is an admission error exactly at the boundary where a worker has the least room to recover.

An allocated backing pool and logical ownership are different ledgers. Returning a page to the free list changes ownership but may leave device-reserved bytes unchanged. The allocator can reuse that page without asking the device runtime for another allocation. Do not use the device allocation graph alone to decide whether request memory was reclaimed.

Pick a reservation policy you can explain

This course reserves the complete declared output allowance at admission. It is conservative: many requests stop early, so some capacity waits unused. Its advantage is a simple promise under the specified workload. A request admitted with a 257-token output limit has room to reach that limit without evicting another running request.

An incremental policy can admit more requests and allocate new pages as outputs grow. It must then define what happens when no page is available: defer a step, preempt a victim, reject growth or reserve emergency headroom. Those choices affect the latency contract. Calling the policy incremental does not make the obligation disappear.

Reserve and admit as one scheduler decision. Two concurrent admissions that both inspect the same free count can each look safe and jointly overcommit. A lock or serialized scheduler can protect the local decision; a stale metric sent to a router cannot provide that guarantee.

Charge shared pages once

Eight requests in one cache trust group can share the 128 complete prefix pages. Each still needs 17 private pages for its 257-token suffix. The total is 264 pages, which exceeds this course's 256-page pool. Without sharing, the same eight requests need 1,160 pages.

Reuse changes the feasible concurrency substantially, yet it does not make eight safe. Seven requests need 128 plus seven times 17, or 247 pages. The remaining nine pages cannot satisfy an eighth private reservation. If every request belongs to a different trust group, the prefixes need separate physical copies and only one 145-page request fits.

This calculation assumes identical semantic input inside each authorized group, no extra speculative slots and no retained unused prefixes. Add those owners explicitly when the assumptions change. A cache entry with no active request may still own memory until eviction.

Do this before moving on

Use the prefix ownership lab with eight requests and a 2,048-token common prefix. Predict the physical allocation for one group and eight groups before changing the control. Then write an admission decision for the 512 MiB pool.

The decision should admit at most seven requests in the shared case and one in the isolated case under full output reservation. State how a shorter output allowance would change the result. The answer earns credit for page rounding and ownership, not merely for quoting a memory total.

Go deeper

• Paged KV memory explains the logical-to-physical mapping behind this ownership ledger.
• Prefix block hashing identifies the semantic conditions required before reuse is safe.
• KV bytes and budget shows where this pool belongs in the full memory inventory.

Key takeaways

• Charge whole pages and include the declared future output.
• Shared storage is counted once; each private suffix still needs a reservation.
• Logical free pages and device-reserved memory answer different questions.

Check yourself

Say or write your answer first, numbers included, then reveal. Producing it from memory is the skill an interviewer is checking.

1. 1How many 16-token pages cover a 2,048-token prompt and 257 generated tokens?

   144145257

2. 2Why do eight shared-prefix requests fail admission into the 256-page pool?

   Answer it, then reveal