The Softmax Behind Sampling

Softmax

Part of: Temperature & Sampling

You've used the word "probability" a lot, blue has 0.60, clear has 0.15. But the model never produces those numbers directly. Inside, it emits raw scores that can be any size: 8.2, -3.7, 0.0, even 50. The function that turns that messy pile of scores into clean probabilities that sum to 1 is softmax , and it's the exact spot where temperature does its work. What it is A model's final layer outputs one logit per token, an unnormalized score. Logits aren't probabilities: they can be negative, they don't sum to anything tidy, and a logit of 4 isn't "twice as likely" as a logit of 2. Softmax is the rule that converts logits into a real probability distribution. It does two jobs at once: it makes every value positive (via exp), and it makes them sum to exactly 1 (by dividing by the total). The headline property: softmax is relative . Only the gaps between logits matter, not their absolute size. Logits [2, 1, 0] and [12, 11, 10] produce the identical distribution, because the differences are the same. How it works Three steps: exponentiate each logit, sum the results, then divide. Temperature slips in before all of that by scaling every logit by 1/T. Because softmax exponentiates, gaps g

Challenge: The Probability Oracle