sglang

A one-layer drafter that beats an 8B model on acceptance sounds wrong until you ask what it is reading. The answer prices the EAGLE head on Llama 3.3 70B, explains the feature trick, and names the three changes in EAGLE-3.

Both reuse KV for shared prefixes, and on a 2K-token chat prompt their hit rates differ by under one percent. The answer finds the differences that do matter, from eviction policy to whether a cache key means anything outside one process.

The GPU finishes a decode step and waits while Python appends tokens, checks stop strings and plans the next batch. The answer shows how the loop reorders that work, where the placeholder tokens live, and the request that pays for it.

Eight tenants, eight adapters, one base model and one batch. The answer explains why the extra cost scales with distinct adapters rather than requests, and which default serializes them.