Run a model and keep its evidence
Run a pinned CPU model, compare cached and full execution, and retain every request outcome. Separate first-execution observations, warm repeats and narrow task checks before building the serving state machine.
45 MIN · PROJECT
By InferenceInterviews · Updated
Course lessons Lesson 4 of 25
TL;DR: Keep exact input IDs, cached-forward checks and complete request records from a local model run. Those artifacts establish a reproducible starting point; the timing boundary and tiny task sample limit what you can conclude from them.
Where you are. You have defined the model identity and measurement contract. This project turns them into files you will reuse through capacity, scheduling and the final experiment review.
Download the starter and establish one boundary
Get the free starter ZIP from the course project downloads. Extract it and read README.md. The starter includes the model runner and workbench, a small attention oracle, visible tests and an incomplete serving engine. The completed engine and annotated walkthrough are a separate Premium download. You can implement and test every starter stage yourself.
Begin with the standard-library checks:
python3 -m unittest test_project.Attention test_workbench -v
The attention check gives a zero query three values: 2, 4 and 9. All scores become zero, so softmax assigns equal weights and returns (2 + 4 + 9) / 3 = 5. The prefix check then compares incremental and full computation. This scalar model makes cache position errors visible before dependencies or a model download enter the experiment.
Follow the README's pinned installation commands for the actual CPU model. Use a new output directory for each attempt. The first run downloads public weights; subsequent runs can use the cached snapshot offline. No API key or GPU is required. Keep installation and model-loading failures separate from measured request failures: a process that never loads a model has not completed a request experiment.
Read the files in causal order
manifest.json identifies the runtime, model files and loading work. first-run.json records the first execution after model loading in that process. Two warm-repeat files follow. The runner performs its correctness probes afterward so those probes do not warm the first recorded execution.
This distinction matters even on a CPU. A new process may reuse files already present in the operating system's cache. Its first execution is therefore not evidence of an uncached machine startup. The manifest keeps snapshot resolution and model load time separate from request timing so you can describe the boundary you actually observed.
The correctness file contains three prefix splits for each of three fixed prompts, giving nine cached/full comparisons. It requires numerical agreement within the declared tolerance and identical greedy choices at every compared position. It also rejects a deliberately duplicated input token and a missing-prefix negative control. These checks cover the fixtures and precision in the manifest. They do not prove equivalent behavior for every context length or model.
Inspect failures before comparing rates
Each request row retains its scheduled arrival, send time, individual token observations, output IDs and terminal status. Here the driver observes token availability after a CPU forward pass and greedy selection. A browser has received nothing at that boundary. An HTTP adapter would need separate client receipt timestamps and transport failure records.
The runner sends the next fixture after the previous one finishes. Its completed requests per second describe this sequential run; they cannot establish how a server behaves under an externally imposed arrival rate. Keep a separate experiment for queueing and capacity.
Open the generated HTML reports, then run the comparison command from the README. The workbench retains unsuccessful requests in the offered total. A synthetic two-second window containing one qualifying completion and one failure has observed attainment 1 / 2 = 0.5 and qualified rate 1 request / 2 seconds = 0.5 requests/s. Dropping the failure would change the first denominator and hide an observed outcome.
Read the actual generated text too. The fixture's task rule is an exact comparison after whitespace trimming and case folding. A helpful but verbose response can fail that formatting contract. These three prompts are too narrow to characterize general quality. Repeating them improves your view of timing variation but does not create independent task samples; the workbench's quality comparison remains inconclusive.
Carry the artifacts into one project
Follow PROJECT.md through reservation, bounded scheduling, cancellation and indexed token emission. Each stage names a test command, a required artifact and a lesson to revisit when it fails. The real model runner establishes the execution contract. The small engine then isolates serving ownership with a scalar worker whose ordering you can inspect.
The final checkpoint feeds a complete synthetic engine trace into the same workbench used for the CPU records. Its event intervals are declared invented arithmetic inputs. Keep the evidence labels intact. The engine has no HTTP transport or GPU buffers, so passing its stream-emission tests establishes neither network delivery nor physical device reclamation.
Memory and cost have similarly specific boundaries. The runner records a process lifetime RSS high-water mark, which includes more than model tensors. It supplies no automatic price for your computer. The workbench leaves cost unknown until you provide a window-wide allocation, and leaves cost per qualified request unknown when quality is unassessed or no request qualifies.
Do this before moving on
Submit the manifest, correctness file, first execution and warm-repeat records, plus one comparison report. Add a short note identifying the earliest boundary each artifact checks. Pick a generated answer that fails the formatting contract and explain why that is different from a numerical cache failure.
Then change one copied synthetic record to contain an unassessed task outcome. Predict which report fields become unknown before running the workbench. Preserve the original record. In a later session, reconstruct the offered-request denominator from the raw rows without looking at the summary.
Go deeper
- Model artifact compatibility explains which identities must agree before you trust a comparison.
- Numerical and batch invariance distinguishes close logits from stable generated-token decisions.
- Cached execution equivalence develops the position and prefix checks beyond this starter's fixed fixtures.
- Cancellation ownership supplies the lifetime invariant for your later project stages.
- Reproducible experiment review turns these records into a controlled configuration comparison.
Key takeaways
- Preserve processed inputs and numerical checks alongside measured outputs.
- Name the observation boundary before interpreting latency or memory.
- Retain failures and unknown quality in the experiment record.
- Repeated prompts do not establish independent task evidence.
Check yourself
Work it out before revealing the explanation. Include the units, assumptions and the condition that would change your answer.
1Nine cached/full probes pass. What claim can you make?
2The first-run file starts after model loading. Does its first-token time include downloading weights?
3One success and one failure occupy a two-second complete window. What are observed attainment and qualified rate?
4Why does an unchanged warm-repeat result not pass a quality release decision?
Sign in to track which lessons you have finished.