Attention in Plain English

Attention

Part of: How AI Actually Works

Read this: "The trophy didn't fit in the suitcase because it was too big." What does "it" mean, the trophy or the suitcase? You knew instantly: the trophy. To predict the next word sensibly, a model has to make the same call, it has to look back at the right earlier words. The mechanism that lets it do that is called attention , and it is the single idea that made modern LLMs work. What it is Attention is how the model decides, for each next-token prediction, which earlier tokens matter most. Not every previous word is equally relevant. When predicting the word after "it was too big," the model needs to weight "trophy" heavily and "suitcase" lightly. Attention assigns each earlier token a relevance weight , then blends them by those weights to inform the prediction. Think of it as the model asking, at every step: given what I'm trying to predict right now, which words in the context should I pay the most attention to? How it works For the token it is currently focused on, the model computes a relevance score against every earlier token. Those scores get turned into weights that sum to 1 (yes, softmax again, from lesson 5). Then the earlier tokens are mixed together in proportion to

Challenge: Attention Focus Finder