Greedy vs Sampled Decoding

Decoding

Part of: Temperature & Sampling

Once softmax hands you a probability distribution, you still have to commit to one token. There are two fundamentally different ways to do that, and the choice quietly decides whether your model is a calculator or a poet. Greedy decoding always grabs the single highest-probability token. Sampled decoding rolls the weighted dice. Same distribution, opposite philosophies. What it is Decoding is the rule that turns a probability distribution into a chosen token at each step. - Greedy decoding: take the argmax : the token with the highest probability, every single time. No randomness. Same input, same output, forever. - Sampled decoding: draw a token at random in proportion to its probability (this is the sampling from Lesson 1, with temperature and top-p shaping the odds first). Different runs can pick different tokens. Greedy is the special case you get when temperature hits 0: the distribution becomes so sharp that the top token has effectively all the mass, so "sample" and "take the max" become the same move. How it works The two strategies diverge in one line of code, argmax versus a weighted draw: Greedy looks attractive, always pick the best!, but "best at each step" is not "bes

Challenge: The Greedy Decoder