Top-p and Top-k

Nucleus sampling

Part of: Temperature & Sampling

Temperature reshapes how lopsided the odds are. But there's a separate problem: even after reshaping, the model's hat still contains tens of thousands of junk tokens, each with a tiny sliver of probability. Add up enough slivers and there's a real chance of drawing garbage. Top-k and top-p fix this by throwing out the junk before the draw. What they are Both are truncation methods, they shrink the pool of tokens you're allowed to sample from. - Top-k: keep only the k highest-probability tokens; discard the rest. With k=3, only the top three are eligible, no matter how the rest look. - Top-p (nucleus sampling): keep the smallest set of top tokens whose probabilities add up to p. With p=0.9, you keep tokens from the top down until their cumulative probability hits 0.9, then stop. The key difference: top-k keeps a fixed count ; top-p keeps a variable count that adapts to how confident the model is. How it works After keeping the chosen tokens, you renormalize : rescale the survivors so they sum to 1 again, then sample. Here's top-p in code: Why top-p adapts: if the model is very sure (one token at 0.95), top-p keeps just that one, tight. If the model is unsure (probabilities spread th

Challenge: The Nucleus Filter