Rule-Based Graders

grader functions for different task types

Part of: AI Evals Harness

"Compare to the expected string" only works when there's one correct answer. A classifier has one right label. A chatbot reply, an extracted number, and a JSON blob each need their own way of deciding pass or fail. That's a grader : a small function with the signature grader(actual, expected) - bool, swapped in per test case. Four graders that cover most tasks - exact match for classifiers and short structured labels. - contains for free-form text where you only need one fact present, like a support reply that has to mention the refund amount somewhere. - in range for numeric extraction, where the model might say "9.8" and you meant "10.0". A small tolerance forgives rounding without letting real errors through. - has keys for JSON-shaped output where you care that the schema was respected, not the exact wording. Routing to the right grader A real test case records which grader it needs, and the runner looks it up: That one dictionary lookup lets a single harness grade a sentiment classifier, a support-reply generator, and a price extractor through the same runner loop. Only the grader entry per case changes. Why this matters A harness that only does exact match will reject perfect

Challenge: Route to the Right Grader