Tokens: The Model Doesn't See Words

Tokens

Part of: How AI Actually Works

The model never sees the word "unbelievable." It sees something more like un, believ, able. Three pieces. These pieces are called tokens , and they're the real units a model reads and writes. What a token is A token is a common chunk of text. Sometimes it's a whole word ("cat"), sometimes part of a word ("ing"), sometimes a space-plus-word (" the"), sometimes just punctuation (","). Models use a fixed vocabulary of roughly 50,000 to 100,000 tokens, learned by finding which chunks show up most often. Rough rule of thumb for English: 1 token ≈ 4 characters ≈ ¾ of a word. So 100 words is about 130 tokens. Don't memorize this. You'll look it up when it matters. Why you should care Tokens aren't trivia. They control money, limits, and quality: - Cost. APIs charge per token, input and output. Longer prompts and longer answers cost more. - Context limits. A model can only "see" so many tokens at once, up to its context window. Stuff too much in and the oldest text falls off the edge. - Weird failures. Tasks like "count the letters in 'strawberry'" trip models up partly because the model sees tokens, not individual letters. It literally isn't looking at the r's the way you are. How splitti

Challenge: The Token Meter