Cutting Costs

Optimization

Part of: Cost & Token Budgeting

You've found where the money goes: long outputs and a transcript that gets re-billed every turn. Now the satisfying part, there are a handful of simple levers that can quietly cut a bill by half or more, without making the product worse. Most teams reach for an expensive model when a few of these would have done it. What it is Four practical levers, roughly in order of how often they help: 1. Trim or summarize history. Don't resend the entire transcript forever. Keep the last few turns, or replace old turns with a short summary. This directly attacks the compounding-input problem from the last lesson. 2. Cap max output. Set a limit on how many tokens the reply can use. Since output is the expensive side, a sensible cap is the highest-impact single setting. 3. Use prompt caching. If the same big chunk (a long system prompt, a fixed document) is sent on every call, providers can cache it so repeats are billed at a steep discount instead of full price. 4. Choose a smaller, cheaper model. When the task is easy, classification, short rewrites, simple extraction, a smaller model often does it fine for a fraction of the price. Save the big model for when quality truly needs it. How it wor

Challenge: The Optimization Audit