The Weighted Report and Regression Gate

combining graders and catching regressions

Part of: AI Evals Harness

A real feature usually gets both kinds of grading at once, a cheap rule check plus a nuanced judge score, and neither one alone tells the whole story. This lesson combines them into one number per case, then compares today's overall number against yesterday's. A single run in isolation can't tell you whether you just made things worse. One composite score per case Blend the rule result and the judge score into a single 0-to-1 composite, weighted by how much you trust each signal: When a case has no judge score, maybe it's cheap enough to skip judging on every run, the composite falls back to the rule result instead of silently zeroing it out. Sampling the judge to control cost Judge calls are extra API calls, and running one on every case on every commit adds up fast on a large test set. A common pattern runs the cheap rule grader on the full set every time, but only runs the judge on a random sample in CI, and on the full set nightly or before a release. You trade a little precision for a much smaller bill on the runs that happen most often. Catching a regression An overall score by itself doesn't tell you whether it's better or worse than last time. Store the last known-good scor

Challenge: Composite Score and Regression Gate