Skip to content

Reproduce the model, not just the file

Pin preprocessing, weights, precision layout and runtime together. A model that loads can still produce different token histories, invalidating cache reuse and performance comparisons.

15 MIN

By InferenceInterviews · Updated

Course lessons Lesson 3 of 24

TL;DR: Reproduce the effective model by pinning preprocessing, weights, execution layout and generation policy together; a readable checkpoint alone proves neither equivalent outputs nor reusable cached state.

Where you are. You have a request trace and benchmark contract. This lesson produces the manifest and probes that make those results reproducible after a migration.

The checkpoint loads, but the request changed

A synthetic migration keeps every weight byte identical. The new tokenizer wrapper adds a beginning-of-sequence token, while the old caller already supplied one. A request that previously became [1, 7, 9] becomes [1, 1, 7, 9]. The new system has a different sequence length, shifted positions and different attention inputs. Neither a matching model name nor a successful warmup reveals the mistake.

Predict the first useful comparison: generated text, final logits, or token IDs? Compare the token IDs first. Final outputs combine many possible differences and can conceal the earliest divergent boundary. Find that boundary before investigating numerical kernels.

The same principle applies to image preprocessing, audio resampling and adapter selection. A model artifact is an execution agreement. Its identity must include the transformations that determine the tensors the weights consume.

Record four boundaries

BoundaryRecordProbe
Input semanticsTokenizer and template revisions, special tokens, media transformsExact IDs or transformed tensors for fixed fixtures
Model representationWeight hashes, config, adapter identity, quantization metadataShapes, layouts and selected numerical references
ExecutionRuntime revision, kernel selection, device and precisionReference forward comparisons across relevant shapes
GenerationSampling transforms, stop policy, RNG ownershipDistribution and termination checks under fixed inputs

A file hash is useful because it identifies bytes. It does not establish that two differently serialized files compute the same function, nor that identical bytes are interpreted under the same layout. Packed low-bit weights need their scale convention, group size and packing order. A runtime can accept a tensor shape while applying the wrong permutation.

rendering diagram…

Design probes that localize a failure

For the duplicate-token case, store the expected IDs and a fingerprint of the template. A short fixture is enough to expose that defect, but it does not test long-context positions. Add a prompt near a position boundary, a non-ASCII string, empty input under the supported API contract, and any media or adapter combination that changes preprocessing.

Compare intermediate outputs selectively. A full-precision reference and an optimized implementation may differ slightly because reductions occur in a different order. Set tolerances appropriate to the tested operation and retain absolute and relative errors, including behavior near zero. A single global tolerance can hide a severe error in small outputs or fail harmless error in large ones. Token agreement is an additional task-level observation; it is not a complete numerical proof.

For stochastic decoding, equal seeds are insufficient when batch reordering changes which request consumes which random draw. Bind randomness to stable request identity and step, then decide whether your contract requires replay within one implementation or equivalence of a probability distribution. Record this choice in the manifest so an experiment cannot quietly switch standards after a mismatch.

Decide whether old cache survives

Cached KV is the result of executing a particular prefix under a particular semantic configuration. A changed adapter, positional scheme or model revision can invalidate it even when the token IDs match. Namespace cache entries by the identity that determines their values. Include authorization scope where reuse crosses tenants; identical text does not grant permission to share private state.

A rolling deployment therefore needs two readiness checks: can the new worker execute the intended model, and can it safely consume any state routed to it? If a transfer format is incompatible, recompute under the new revision or route the request to its original pool. Never reinterpret old cache merely because the byte count fits.

These boundaries constrain packaging decisions. Containers, exported graphs and compiled artifacts improve repeatability only when their dependencies and compatibility are recorded. They do not replace semantic probes. Keep the cold-start stages visible too: fetching bytes, allocating tensors, compilation and warmup can have different failure modes.

Do this before moving on

Write a manifest for the synthetic tokenizer migration. Include one exact-input fixture, one numerical probe and one stop-policy test. Then change only the adapter revision. Mark which fields and cache namespaces must change, and which benchmark inputs should remain fixed for a fair comparison.

Reconstruct the probe order from memory in a later session. Explain why you would stop at an input-ID mismatch instead of collecting a large latency benchmark first.

Go deeper

Key takeaways

  • Identify the earliest divergent boundary before explaining final output differences.
  • Pin transformations and layouts alongside weight bytes.
  • State the numerical and stochastic correctness contract explicitly.
  • Cache compatibility follows model semantics and authorization scope.

Check yourself

Work it out before revealing the explanation. Include the units, assumptions and the condition that would change your answer.

  1. 1Weights match but token IDs differ. Which investigation comes first?

  2. 2Two packed tensors have equal shapes and byte counts. Does that establish compatibility?

  3. 3Only an adapter revision changes. Can identical token prefixes reuse the previous adapter's KV?

  4. 4Why keep a near-zero numerical probe as well as a large-output probe?

Sign in to track which lessons you have finished.