Why the Same Prompt Gives Different Answers

Sampling

Part of: Temperature & Sampling

Ask a model "write me a one-line slogan for a coffee shop" twice and you'll often get two different lines. Same prompt, same model, different answer. That isn't a bug or a hidden mood, it's the model doing exactly what it always does: rolling dice over its own predictions. What it is From an earlier lesson you know a model predicts the next token. But it doesn't predict a single token, it predicts a probability distribution : a score for every token in its vocabulary, all adding up to 1. For the prompt "The sky is" it might assign blue a 0.60 chance, clear 0.15, dark 0.10, and tiny slivers to thousands of others. Then comes the step people miss. The model does not have to take the top one. It samples : it picks a token according to those probabilities , like drawing a weighted ticket from a hat. blue is the fat favorite, but clear and dark are still in the hat. Draw again on a different run and a different ticket can come out. How it works The randomness is deliberate. If a model always took the single highest-probability token (that strict mode is called greedy decoding ), it would be perfectly repeatable, and often dull, repetitive, and prone to getting stuck in loops. Sampling a

Challenge: The Weighted Draw