Repetition and Frequency Penalties

Penalties

Part of: Temperature & Sampling

Even with sampling and a sensible top-p, long generations have a nasty habit: they repeat. The model says "very, very, very" or circles back to the same sentence, because once a token appears it often becomes more likely to appear again. Repetition penalties fight this directly by docking the logits of tokens the model has already used, before sampling. Two flavors do the work: presence and frequency . What it is A penalty is a number subtracted from a token's logit based on its history : how it has already been used in this generation. - Presence penalty: a flat subtraction applied once to any token that has appeared at all , no matter how many times. It nudges the model toward bringing in new topics and words. - Frequency penalty: a subtraction that scales with the count : the more often a token has appeared, the harder it gets pushed down. It specifically punishes overusing the same token. Both leave never-seen tokens untouched. They only reshape the odds for words already on the page. How it works You apply the penalties to the logits before softmax. For each candidate token, look up its count in the history and adjust: With a frequency penalty of 0.5 and a count of 3, the lose

Challenge: The Penalty Engine