Scoring Fuzzy Answers

Scorers

Part of: Fine-Tuning & Evals

Your model answers "The capital of France is Paris." Your reference says "Paris." Exact match scores that wrong . The answer is perfect. The scorer is dumb. Picking the right scorer is most of the work in evals, get it wrong and your number lies to you in both directions. What it is A scorer is the function that turns a model output and a reference into a score. There's a whole ladder of them, ordered from strictest to loosest: - Exact match : output == expected. One acceptable string: a label, an ID, a yes/no. Brutal but unambiguous. - Substring / contains : did the output include the required answer? expected.lower() in output.lower(). This rescues the Paris case. Great when the model is allowed to be chatty as long as it says the right thing. Watch out: "not Paris" also contains "Paris," so a contains scorer can be fooled by negation. - Similarity : for free-form text where wording varies, score how similar two strings are. Jaccard similarity (word overlap) is the cheap, dependency-free version; cosine similarity over embeddings is the production version. Closer meaning, higher score. How it works Jaccard similarity is the size of the word-set intersection over the size of the u

Challenge: The Jaccard Scorer