The Shape of an Eval

test cases and a scored loop

Part of: AI Evals Harness

Ship an AI feature and someone will ask "does it actually work?" Eyeballing five examples in a notebook doesn't answer that at scale. An eval harness does: you run a test set through your feature, grade every result automatically, and "does it work" becomes a number you can watch instead of a feeling you have to trust. What we're building By lesson 8 you have a script. Point it at a test set and a function that calls your AI feature, and it prints a scored report with no manual review. It has three parts. A test set holds inputs plus what a correct answer looks like. A grader decides pass or fail for one case. A runner loops the test set through your feature and the grader, then tallies the results. The anatomy of a test case The smallest useful test case is an input and an expected outcome: In a real harness, "input" gets sent to your actual AI feature: The harness compares that output to expected, case by case, and records a pass or a fail. Why this matters Prompts drift. You tweak a system prompt to fix one bad answer and quietly break two others you never re-checked. Without a harness you find out when a user complains. With one you re-run the whole test set in seconds and see

Challenge: Run the Test Set