LLM-as-Judge

using a model to grade subjective quality

Part of: AI Evals Harness

Some outputs have no single correct string to compare against. "Is this customer reply polite and helpful?" isn't an exact-match question. For tasks like that, you grade with a second model call, the judge . You show it the input, the output, and a rubric, and ask it to return a structured verdict. The judge prompt A judge system prompt pins down what to score and what shape to return, the same discipline as any other structured-output prompt: Parsing the verdict defensively The judge is still an LLM, so its reply can arrive with a stray preamble or a code fence around the JSON, like any other structured-output call. You extract and validate it the way you would any model JSON. You also check the score is in range, because a judge that returns "score": 9 on a 1-5 scale gave you a broken verdict, not a real one: Why this matters, and its limits An LLM judge lets you grade tone, helpfulness, or faithfulness to a source document, things a rule can't cleanly check. But the judge is not ground truth. It can be miscalibrated, inconsistent, or fooled by a confident-sounding wrong answer. Treat it as a second opinion you spot-check against a handful of human ratings now and then, not an or

Challenge: Parse the Judge's Verdict