Quantify what continuous batching buys over static batching, then pick the chunked-prefill token budget and defend it.

TL;DR: Continuous batching recovers idle slots, but its wall-clock gain depends on step cost, admission and the request mix. A finite two-slot example takes 23 static steps versus 17 with refill, a 1.35x improvement only if steps cost the same; the weight-only H100 SXM BF16 ridge near 295 tokens is a starting point for a token-budget sweep, not a serving default.

How to approach it

Ask for output lengths, arrivals, available KV memory and the TTFT/ITL targets. Separate slot utilization from elapsed time: refilling a slot changes the batch's attention work and may add prefill. Then distinguish total scheduled tokens from prefill tokens. A budget consumed by decoders cannot simultaneously serve new prompts.

A strong answer

For independent static batches with $B$ occupied slots, each batch performs $\sum_i L_i$ useful token-steps while reserving $B\max_i L_i$ slot-steps. Across many batches, the ratio of totals converges to:

$$U_{\mathrm{static}} = \frac{\mathbb{E}[L]}{\mathbb{E}[\max_{i\le B} L_i]}$$

This is a ratio of expectations, not the expected utilization of one batch. With independent, continuous exponential lengths, $\mathbb{E}[\max L]=\mu H_B$. That analytically convenient distribution is not a claim about production output lengths, which are integer-valued, truncated and workload-dependent.

B = 32: H_32 = sum(1/i, i=1..32) = 4.05850
long-run static slot utilization = 1 / H_32 = 24.6397%
ideal full-slot/refill ratio     = H_32      = 4.05850x
assumptions: constant step duration, saturated backlog, cost-free admission,
             no prefill, memory limit, prefix effect or scheduler overhead
sanity: identical lengths give utilization 1 and no slot-recovery benefit

Iteration-level scheduling admits replacements between iterations; Orca, OSDI 2022, provides the mechanism. Its measured speedups are results for its workloads and systems, not consequences of the harmonic-number calculation.

Run this finite-queue countercheck with Python 3, no dependencies. All requests arrive at time zero, lengths count decode steps, and a completed slot refills on the next step:

from heapq import heapify, heappop, heappush
from math import fsum

def schedule(lengths, slots):
    if slots < 1 or any(x < 1 or int(x) != x for x in lengths):
        raise ValueError("positive integer lengths and slot count required")
    static = sum(max(lengths[i:i + slots]) for i in range(0, len(lengths), slots))
    finish = [0] * slots
    heapify(finish)
    for length in lengths:
        heappush(finish, heappop(finish) + length)
    return static, max(finish)

lengths = [2, 6, 3, 8, 2, 4, 5, 2]
a, b = schedule(lengths, 2)
print(f"static {a}; continuous {b}; equal-step ratio {a / b:.4f}")
print(f"useful token-steps {sum(lengths)}; H32 {fsum(1/i for i in range(1, 33)):.5f}")

static groups       [2,6] [3,8] [2,4] [5,2]
static steps        6 + 8 + 4 + 5 = 23
refill slot paths   2+3+8+2=15; 6+2+4+5=17
useful work         2+6+3+8+2+4+5+2 = 32 token-steps
static utilization  32 / (2 x 23) = 69.57%
refill utilization  32 / (2 x 17) = 94.12%
equal-step speedup  23 / 17       = 1.353x

The slot paths above must follow arrival order: the executable schedule is the reference for ties. Real step times vary with active tokens, context lengths, graph padding and prefill admission. Measure elapsed time and SLO attainment; do not multiply server throughput by the slot ratio.

For a synthetic dense model with 8 billion multiplied parameters and 16 GB of streamed BF16 weights, the vendor-reported H100 SXM dense BF16 peak gives this weight-only approximation:

weight read = 16e9 / 3.35e12                         = 4.776 ms
linear work/token = 2 x 8e9 / 989e12                 = 16.178 us
ridge = 4.776 ms / 16.178 us                         = 295.224 tokens
step(T) = max(4.776 ms, T x 16.178 us)
prefill rate with 64 decoders = (T - 64) / step(T), for T > 64

Even this toy does not make prefill throughput flat at the ridge when a fixed decode population consumes part of the budget. Attention, activation traffic and imperfect GEMMs are absent. The actual knee can move in either direction as effective compute and bandwidth change. The H100 specifications supply the peaks, checked October 17, 2026; the table is analytical, not a benchmark.

Sweep budgets with the same arrival trace, length mix, cache state and concurrency limits. Record queue wait, uncached prefill tokens/s, p99 ITL, preemptions and completed requests meeting both latency targets. Pick the budget with the best goodput within the contract. Larger budgets can be appropriate for offline work; a tight streaming SLO can justify operating below a throughput knee.

What interviewers probe next

"An 8,192-token prompt, budget 2,048, 64 decoders throughout?" Prefill gets 1,984 tokens per step, so it needs $\lceil8192/1984\rceil=5$ steps before runtime and queueing effects.

"All outputs have the same length." Static batches have no early-finish holes; continuous admission can still change waiting time under staggered arrivals.

"The larger budget improves tokens/s but hurts p99." Compare goodput and rejection rates on the same offered load before accepting the change.

How it's scored

Weak: promises a fixed several-fold speedup or fills every step to the largest possible budget.

Hire: derives slot utilization, states the constant-step assumptions and subtracts decode tokens from the prefill allowance.

Strong Hire: distinguishes ratio-of-totals from per-batch averages, catches the finite drain, and chooses a measured operating point by SLO-qualified completions.

Common mistakes

• Calling the harmonic-number ratio a measured production speedup.
• Treating a finite queue as a permanently saturated one.
• Labeling all tokens in a mixed step as prefill throughput.
• Inferring a tail-latency guarantee from an average step-time model.