Skip to content
← Serving Engine Internals / 17
hardNewMistral AICerebras

An RL trainer pushes new weights into a running vLLM engine. What happens to the sequences and KV already in flight?

Moving 16 GB of weights takes a third of a second. Deciding what to do with 64 half-finished rollouts costs up to 100 times that. The answer walks the engine's pause modes, prices each one, and shows what the per-token weight-version spans do and do not record.

Updated Nov 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

Moving 16 GB of weights takes a third of a second. Deciding what to do with 64 half-finished rollouts costs up to 100 times that. The answer walks the engine's pause modes, prices each one, and shows what the per-token weight-version spans do and do not record.

10 answers per topic instead of 5, plus saved progress and bookmarks · costs nothingor unlock all 213 remaining answers · $25

Every worked answer and derivation in all 12 topics, every concept page, the full practice-test bank, and every code drill’s edge-case tests and reference solution.

6 months · One payment · No auto-renewal

UP NEXT ON YOUR JOURNEY
Next in this trackSame KV pool: vLLM wins TTFT, SGLang never freezes a stream, and P99 ITL barely differs. Explain it and fix both.Next in this trackServe Llama 3.1 8B at 256K when its config stops at 128K: what changes, what breaks silently, and how do you prove it?Popular right nowWhy is batch-1 decode on a 70B model bandwidth-bound, and what is its latency floor on two H100s?
DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.