Reproduce the model, not just the file
Pin preprocessing, weights, precision layout and runtime together. A model that loads can still produce different token histories, invalidating cache reuse and performance comparisons.
15 MIN
By InferenceInterviews · Updated
Course lessons Lesson 3 of 24
TL;DR: Reproduce the effective model by pinning preprocessing, weights, execution layout and generation policy together; a readable checkpoint alone proves neither equivalent outputs nor reusable cached state.
Where you are. You have a request trace and benchmark contract. This lesson produces the manifest and probes that make those results reproducible after a migration.
The checkpoint loads, but the request changed
A synthetic migration keeps every weight byte identical. The new tokenizer wrapper adds a beginning-of-sequence token, while the old caller already supplied one. A request that previously became [1, 7, 9] becomes [1, 1, 7, 9]. The new system has a different sequence length, shifted positions and different attention inputs. Neither a matching model name nor a successful warmup reveals the mistake.
Predict the first useful comparison: generated text, final logits, or token IDs? Compare the token IDs first. Final outputs combine many possible differences and can conceal the earliest divergent boundary. Find that boundary before investigating numerical kernels.
The same principle applies to image preprocessing, audio resampling and adapter selection. A model artifact is an execution agreement. Its identity must include the transformations that determine the tensors the weights consume.
Record four boundaries
| Boundary | Record | Probe |
|---|---|---|
| Input semantics | Tokenizer and template revisions, special tokens, media transforms | Exact IDs or transformed tensors for fixed fixtures |
| Model representation | Weight hashes, config, adapter identity, quantization metadata | Shapes, layouts and selected numerical references |
| Execution | Runtime revision, kernel selection, device and precision | Reference forward comparisons across relevant shapes |
| Generation | Sampling transforms, stop policy, RNG ownership | Distribution and termination checks under fixed inputs |
A file hash is useful because it identifies bytes. It does not establish that two differently serialized files compute the same function, nor that identical bytes are interpreted under the same layout. Packed low-bit weights need their scale convention, group size and packing order. A runtime can accept a tensor shape while applying the wrong permutation.
Design probes that localize a failure
For the duplicate-token case, store the expected IDs and a fingerprint of the template. A short fixture is enough to expose that defect, but it does not test long-context positions. Add a prompt near a position boundary, a non-ASCII string, empty input under the supported API contract, and any media or adapter combination that changes preprocessing.
Compare intermediate outputs selectively. A full-precision reference and an optimized implementation may differ slightly because reductions occur in a different order. Set tolerances appropriate to the tested operation and retain absolute and relative errors, including behavior near zero. A single global tolerance can hide a severe error in small outputs or fail harmless error in large ones. Token agreement is an additional task-level observation; it is not a complete numerical proof.
For stochastic decoding, equal seeds are insufficient when batch reordering changes which request consumes which random draw. Bind randomness to stable request identity and step, then decide whether your contract requires replay within one implementation or equivalence of a probability distribution. Record this choice in the manifest so an experiment cannot quietly switch standards after a mismatch.
Decide whether old cache survives
Cached KV is the result of executing a particular prefix under a particular semantic configuration. A changed adapter, positional scheme or model revision can invalidate it even when the token IDs match. Namespace cache entries by the identity that determines their values. Include authorization scope where reuse crosses tenants; identical text does not grant permission to share private state.
A rolling deployment therefore needs two readiness checks: can the new worker execute the intended model, and can it safely consume any state routed to it? If a transfer format is incompatible, recompute under the new revision or route the request to its original pool. Never reinterpret old cache merely because the byte count fits.
These boundaries constrain packaging decisions. Containers, exported graphs and compiled artifacts improve repeatability only when their dependencies and compatibility are recorded. They do not replace semantic probes. Keep the cold-start stages visible too: fetching bytes, allocating tensors, compilation and warmup can have different failure modes.
Do this before moving on
Write a manifest for the synthetic tokenizer migration. Include one exact-input fixture, one numerical probe and one stop-policy test. Then change only the adapter revision. Mark which fields and cache namespaces must change, and which benchmark inputs should remain fixed for a fair comparison.
Reconstruct the probe order from memory in a later session. Explain why you would stop at an input-ID mismatch instead of collecting a large latency benchmark first.
Go deeper
- Model artifact compatibility separates file readability, runtime acceptance and semantic agreement.
- Migration with duplicate special tokens provides the concrete debugging sequence for changed input semantics.
- Request randomness across reordering tests reproducibility when scheduler slots are unstable.
- Nondeterminism and batch invariance explains why numerically close logits can change generated tokens.
Key takeaways
- Identify the earliest divergent boundary before explaining final output differences.
- Pin transformations and layouts alongside weight bytes.
- State the numerical and stochastic correctness contract explicitly.
- Cache compatibility follows model semantics and authorization scope.
Check yourself
Work it out before revealing the explanation. Include the units, assumptions and the condition that would change your answer.
1Weights match but token IDs differ. Which investigation comes first?
2Two packed tensors have equal shapes and byte counts. Does that establish compatibility?
3Only an adapter revision changes. Can identical token prefixes reuse the previous adapter's KV?
4Why keep a near-zero numerical probe as well as a large-output probe?
Sign in to track which lessons you have finished.