LLM-as-Judge

LLM Judge

Part of: Fine-Tuning & Evals

Some answers can't be string-matched. "Was this customer reply empathetic?" "Is this summary faithful to the source?" There's no reference string to compare against, empathy isn't a substring. So you do the thing that sounds like cheating but works: you ask a model to grade the output. What it is An LLM-as-judge is a second model call whose only job is scoring. You give it the input, the model's output, and a rubric : explicit scoring criteria, and ask for a score. Done well, it correlates surprisingly closely with human raters, and it runs in seconds across thousands of cases where hiring humans to grade would cost a fortune and take days. The whole game lives in the rubric. A vague prompt ("rate this 1-10") gives noisy, drifting scores that change run to run. A sharp rubric with explicit per-level criteria and a forced output format gives stable, parseable ones. How it works A good judge prompt has four properties: - Crisp criteria. Spell out what each score means: "1 = factually wrong, 3 = partially correct, 5 = fully correct and complete." Don't make the judge invent the scale. - Forced structured output. Demand JSON or a single token so you can parse it. Free-form judge prose

Challenge: Parsing the Judge