Scoring the Reviewer

precision and recall against seeded bugs

Part of: AI Code Reviewer

Every change to the prompt, the rules, or the model changes what the reviewer reports. Without a score you are judging those changes by reading a few reviews and going with your gut. This lesson builds the evaluation that turns "it seems better" into numbers you can compare. A labeled set of seeded bugs. Write a handful of snippets and seed them with known problems. For each one, record the expected findings as (line, category) pairs. That list is the ground truth . Run the reviewer on the same snippets and compare its findings to the labels. - A finding that matches a label is a true positive (TP). - A finding with no matching label is a false positive (FP), noise the user has to read past. - A label with no matching finding is a false negative (FN), a real bug the reviewer missed. Precision and recall. Two numbers summarize the counts. Precision is how much of what the reviewer said was right. Recall is how much of what was there it found. A reviewer that flags every line has perfect recall and terrible precision. A reviewer that reports one sure thing has high precision and poor recall. F1 combines them as 2 p r / (p + r) when you need a single number. When there are no findings

Challenge: Grade the Reviewer