Trimming and Summarizing Old Turns

keeping history in budget

Part of: Persona Chatbot

Now that you can measure a conversation, you can keep it from blowing the budget. When history grows too big, you have two tools: trim it or summarize it. Real chatbots use both. Strategy 1: sliding window The simplest fix: keep only the most recent turns and drop the oldest. It's cheap, predictable, and the cost is bounded, you always know your worst case. The bot forgets old details but recent context stays sharp, which is what most questions need. Two guardrails live in that loop. len(kept) 1 keeps at least one turn so you never send an empty history. And you only ever pop from the front, never touch the system prompt, which lives in its own channel and is never in this list. Strategy 2: summarize-and-drop A sliding window throws old details away. When those details still matter over a long chat, summarize instead: when history gets long, ask the model to compress the oldest turns into a short paragraph, then replace those turns with a single summary turn. It costs an extra API call and a little latency, but a 100-turn conversation collapses to a paragraph plus the last few turns and still remembers your name. When to use which Start with a sliding window: it's free and bounded,

Challenge: Summarize-and-Drop