Tracking Evals Over Time

Tracking

Part of: Fine-Tuning & Evals

A single eval score is a snapshot. The real value shows up when you have a hundred of them, lined up by version, and you can finally answer the question every AI team argues about: "did we actually get better, or does it just feel that way?" A prompt change here, a model upgrade there, a new few-shot example, without a record, you're relitigating the same debate every week. Tracking turns scattered runs into a trend line. What it is Eval tracking is recording every eval run, tagged with what produced it, so you can compare scores across versions over time. Each entry is a row: which prompt version or model version was tested, when, and what it scored. Together they form a regression dashboard : a history that shows whether each change moved the number up, down, or nowhere. The unit of comparison is the version . You change one thing (the prompt, the model, the temperature), re-run the same frozen eval set, and log the result against that version label. Same yardstick, different version, that's what makes the comparison fair. How it works You keep a list of runs in order and walk it, comparing each score to the one before. A drop from one version to the next is a regression : the th

Challenge: The Regression Dashboard