Build an Eval Suite

Eval Suite

Part of: Fine-Tuning & Evals

One eval is a test. An eval suite is your safety net, the thing you run before every prompt change so you catch regressions before users do. The first time it stops you from shipping a "harmless" prompt tweak that quietly broke 8% of your answers, you'll wonder how you ever worked without it. Let's wire the pieces from this module into one runnable harness. What it is An eval suite is a collection of scored test cases run together, reporting per-case results and an aggregate. A single number hides where you broke; the suite shows you exactly which inputs regressed. That's the difference between "something got worse" and "the negation cases broke." A solid suite has four moving parts: - Cases : each with an input, an expected reference, and which scorer to use. - A runner : loops cases, calls the model, applies the right scorer. - Per-case output : pass/fail with actual vs. expected, for debugging. - An aggregate : the pass rate , the headline number you track over time. How it works This pulls exact-match and contains scoring into one runner, with a stubbed model so it's fully deterministic. The math case passes (exact "4"). Paris passes (contains, even though the answer is chatty)

Challenge: The Eval Suite Harness