Caching to Cut Cost

Caching

Part of: Cost & Token Budgeting

You ship a support bot with a 2,000-token system prompt: rules, tone, a product FAQ. Every single message resends that same 2,000 tokens as input. Across a million messages you pay full price for the identical text two million... no, a billion times. Prompt caching exists to stop exactly this waste - and it can cut a repeated prefix's cost by 90% . What it is Prompt caching lets the provider store a fixed prefix of your input - usually the long, unchanging part - and bill it at a steep discount on later calls instead of full price. The discount is large: a cached token often costs around 10% of the normal input rate . The catch is that only a cacheable prefix qualifies: the part of the prompt that is byte-for-byte identical, sitting at the very front, call after call. How it works You mark the stable front of the prompt - the system prompt, a fixed document - as cacheable. The first call pays full price (and sometimes a small write fee) to populate the cache. Every later call that starts with the exact same prefix reads it at the discounted rate, and only the new part (the user's actual question) is billed at full input price: The bigger and more repeated the prefix, the more cachi

Challenge: The Cache Ledger