Tokens: How Models Read and Bill

Tokens

Part of: Prompt Engineering

Paste "ChatGPT" into a tokenizer and it splits into "Chat" + "GPT", two tokens rather than one word. Models don't read words, they read tokens : chunks of text that are usually a few characters long. "cat" is one token. "unbelievable" might be three ("un", "believ", "able"). A space is often glued to the front of the next token. This sounds like trivia until you realize tokens are how you get billed and how the model's memory limit is measured. Every dollar and every length limit in LLM work is denominated in tokens. What a token is A token is a common chunk of text drawn from the model's fixed vocabulary of roughly 50,000 to 100,000 entries. Sometimes a token is a whole word, sometimes a word-piece ("ing"), sometimes a space-plus-word (" the"), sometimes a single punctuation mark. The vocabulary is learned by finding which chunks appear most often, so frequent text packs into few tokens and rare text shatters into many. A rough rule of thumb For typical English, 1 token ≈ 4 characters ≈ 0.75 words . So 1,000 tokens is roughly 750 words, about a page and a half. Don't memorize exact splits. You'll look them up or count them with a tool. Just internalize the rough conversion so you

Challenge: Context Window Budgeter