Evals in CI

CIEvals

Part of: Deploying & Monitoring LLM Apps

You tweak one line of your prompt to fix a formatting bug. It ships. Two days later you discover the tweak quietly broke 12 percent of answers on a different task that you never thought to re-test. With normal code, a unit test would have caught the regression in seconds. With prompts, most teams have no test at all, they ship on vibes. Evals are how you stop doing that. What it is An eval is an automated test for an LLM's behavior. Instead of asserting add(2,3) == 5, you run the model on a fixed set of example inputs, score each output against a rule or expected answer, and compute a pass rate. Run that whole suite automatically on every change, in CI , your continuous integration pipeline, and a prompt change that drops the score below a threshold fails the build before it reaches users. The hard part is that LLM output is not exactly equal to a golden string. So evals use graders : checks like "does the output contain the required field," "is it valid JSON," "does it match this regex," or even "does a second model judge it as correct." How it works A minimal eval suite is a list of cases plus a grader plus a threshold: In CI, raise SystemExit makes the pipeline red, which blocks

Challenge: The CI Eval Gate