Skip to content

Batch encoders by work and state

Encoder batching must account for padded work, deadlines and transient activations. Multimodal requests also need a separate ledger for encoder state, visual tokens and decoder cache.

20 MIN

By InferenceInterviews · Updated

Course lessons Lesson 19 of 24

TL;DR: Batch encoder requests by padded work and deadline slack, and budget multimodal encoder state separately from decoder KV; equal request counts or raw token totals can hide the dominant allocation.

Where you are. You can schedule autoregressive work. Adapt the ledger to embeddings, encoder-decoder models and vision-language requests.

One long sequence changes the whole dense batch

A synthetic encoder receives lengths [16, 16, 16, 128]. A dense padded implementation executes four sequences at length 128. Its linear position count is 4 × 128 = 512, while useful positions total 16 + 16 + 16 + 128 = 176, a padding factor of about 2.91.

For a dense self-attention operation whose score work scales with squared length, the padded term is 4 × 128² = 65,536. The sum of individual terms is 3 × 16² + 128² = 17,152, a factor of about 3.82. These are shape-derived work proxies, not measured latency ratios. Kernel efficiency and other layers still matter.

Grouping the three short requests separately from the long request removes this padding under the assumed implementation. It also creates two launches or batches and may reduce matrix efficiency. A variable-length kernel can change the tradeoff again. Measure the actual execution path rather than treating the padding ratio as a promised speedup.

rendering diagram…

Batching delay spends the same deadline as execution

Suppose a short request has 25 ms remaining until its response deadline. A candidate short batch costs a synthetic 8 ms of execution and 3 ms of downstream work. The batcher can spend at most 14 ms waiting under that estimate. Waiting 20 ms for a fuller batch misses the deadline even if its device throughput improves.

Account for tokenization and preprocessing before forming the device batch. Long raw text, image decoding or audio feature extraction can delay admission independently of accelerator utilization. A token-count limit applied only after preprocessing cannot bound work already queued in that earlier stage.

Embedding services also need a correctness contract for pooling, normalization and truncation. A faster implementation that silently changes truncation length or omits normalization may return a vector of the expected shape while changing its meaning. Keep fixed semantic fixtures alongside shape and latency tests.

A vision request owns more than text KV

A vision-language path can include image decoding, resizing or tiling, an image encoder, projection into the language model's representation, and autoregressive decoding. The image encoder may allocate a large temporary activation peak before decoder KV grows. A free KV pool does not prove the image stage fits.

Write separate lifetimes for raw or preprocessed media buffers, encoder activations, retained encoder outputs and decoder cache. Some architectures insert visual tokens into a decoder sequence; others retain separate cross-attention state. Derive from the actual model graph. “One image” is not a constant token or memory unit when resolution and tiling vary.

Caching encoder outputs requires identity for media content, preprocessing and encoder/projector revisions. Reusing by filename or URL alone is unsafe if content or transforms can change. Authorization scope applies to cached media representations just as it does to text state.

A disaggregated encoder can remove a transient peak from the language-model worker, but introduces a transfer and readiness boundary. Price the representation bytes that actually cross it and keep failed preprocessing or encoder requests in the service denominator.

Bound state across architecture families

An encoder-decoder model may retain encoder outputs while its decoder generates. A hybrid model may carry recurrent state as well as attention cache. Neither can be budgeted accurately by copying the dense decoder's per-token KV formula unchanged.

The reusable method is to name each state object's shape, precision, owner and release event. For an embedding request, output delivery may end the model-state lifetime. For a long speech transcription or multimodal generation, encoder-derived state can survive many decoder steps. Admission must cover the overlap, not merely the largest object considered in isolation.

Do this before moving on

Replace the long sequence with length 64. Recompute linear and squared-length padding factors. Then let the short request's remaining deadline fall to 10 ms with the same 8 ms execution and 3 ms downstream estimate. Decide whether any positive batch wait is feasible.

Draw a lifetime table for a request containing two differently sized images and a text prompt. Mark which bytes belong to the encoder peak and which persist through generation. List the identity fields required to reuse an encoded image safely.

Go deeper

Key takeaways

  • Estimate padded linear and attention work separately.
  • Batch waiting consumes the same response deadline as execution.
  • Track encoder peaks and persistent multimodal state outside the text KV ledger.
  • Cache identity includes media transforms and model revisions.

Check yourself

Work it out before revealing the explanation. Include the units, assumptions and the condition that would change your answer.

  1. 1For [16,16,16,64], what are the padding factors?

  2. 2With 10 ms remaining and 8 + 3 ms of required work, may the batcher wait?

  3. 3Why can an image request OOM when decoder KV has free blocks?

  4. 4Is an image URL sufficient cache identity?

Sign in to track which lessons you have finished.