Harden: Flaky Judges and Retries

majority vote and graceful degradation

Part of: AI Evals Harness

The judge is itself an LLM, so it inherits every unreliability you've learned to guard against elsewhere. It can time out, return malformed JSON, or disagree with itself on two runs of the exact same case. A harness that trusts a single judge call at face value produces a noisy score nobody trusts. Two hardening moves fix that. Retry, and record failure as its own outcome If a judge reply fails to parse, don't silently count it as a fail, and don't crash the whole run. Retry it a couple of times, and if it still won't parse, record a distinct judge error so you can tell "the feature failed" apart from "the judge glitched": Vote out the noise Even a well-formed judge can wobble between a 3 and a 4 on the same input across runs, especially near your pass threshold. Instead of trusting one call, run the judge a handful of times and take the median . The median barely moves when one call comes back an outlier, where an average would drag toward it: If every attempt for a case comes back None, that case gets "NO VERDICT". That's a visible signal the judge is broken for this input, worth investigating on its own rather than as a quality failure. Why this matters An eval harness that cras

Challenge: Vote Out the Noise