Logprobs and Confidence

Logprobs

Part of: Reading Model Outputs

The model wrote "The capital of Australia is Canberra" and it's right, but was it sure ? There's a number hiding in the response that answers exactly that: for every token it generated, the model can report how likely it thought that token was. Those numbers are logprobs , and they're the closest thing you get to a confidence meter on a model's output. What it is A logprob is the log of the probability the model assigned to a token when it chose it. Probabilities run from 0 to 1; their logs run from negative numbers up to 0. A logprob of 0 means probability 1 (dead certain); a very negative logprob like -5 means the model thought that token was unlikely yet picked it anyway. Closer to 0 means more confident; more negative means less confident. Why logs instead of plain probabilities? Probabilities of long sequences get vanishingly tiny, and logs turn awkward multiplication into easy addition while keeping the numbers in a readable range. You don't have to love the math, you just have to read the rule: higher (closer to zero) logprob = the model was more sure. How it works When you request logprobs, each token in the output comes with its score. You scan them for low-confidence toke

Challenge: Confidence Auditor