When You Run Out

Truncation

Part of: The Context Window

Halfway through a long chat, the model suddenly forgets your name, even though you told it ten minutes ago. It isn't being rude or buggy. Your name slid off the desk. This is what running out of context looks like, and it happens in one of two ways. What it is When a request would exceed the window, you hit the ceiling, and there are only two possible outcomes: - A hard error. Some APIs simply refuse: the request is too big, so the call fails with a "context length exceeded" message. Annoying, but at least it's loud. - Silent truncation. Many chat apps quietly drop the oldest text to make the new request fit. No error, no warning, the start of the conversation just disappears. This is the dangerous one, because it fails quietly. Truncation almost always trims from the oldest end. The most recent messages are kept; the beginning falls away. That's exactly why a model "forgets" the early parts of a long chat: those tokens were cut to make room. How it works Picture the conversation as a list of turns. When the total tokens exceed the window, the app pops turns off the front until it fits: Notice the system prompt and your newest message are kept; it's the middle-aged history that get

Challenge: Sliding-Window Truncator