Your First Pass Rate

aggregate scoring and a threshold gate

Part of: AI Evals Harness

A dozen PASS/FAIL lines is progress, but nobody wants to read a hundred of them to decide whether a feature is ready. The next step turns those lines into one number, a pass rate , and one decision, a gate . Did the feature clear the bar or not. From lines to a number Once every case has a boolean result, the pass rate is arithmetic: sum(results) works because True counts as 1 and False as 0 in Python, so summing a list of booleans gives you the pass count directly. The gate A pass rate alone doesn't tell you whether to ship. You need a threshold , the minimum rate you've decided is acceptable for this feature. Below it, the build fails. At or above it, it ships. This is the same idea as a unit test suite failing the build when coverage drops. The assertion here is just fuzzier. It's model output compared to expected output, aggregated across many cases instead of one exact assert. Why a number, not a vibe "It seemed fine when I tried it" doesn't survive a prompt change six weeks later. A pass rate does. Run the harness, get 82%, change the prompt, run it again, get 71%, and you know immediately that something broke, without re-reading a single transcript by hand. The threshold tur

Challenge: Gate the Build