Prompt Caching

Caching

Part of: Streaming, Latency & Caching

You send a 4,000-token system prompt and a long document to a model, then ask ten follow-up questions about it. Without caching, the model re-processes those 4,000 tokens ten times and you pay for them ten times. With prompt caching , it processes them once, stores the result, and the next nine calls reuse it, often 90 percent cheaper and noticeably faster. The trick is structuring the prompt so the cache can actually hit. What it is Prompt caching stores the model's internal processing of a fixed prefix of your prompt so that repeated requests sharing that prefix skip the work. The expensive part of inference is reading the prompt (the "prefill"); caching saves that prefill for the parts that do not change between calls. A cache entry is keyed on the exact prefix of tokens. Reuse only happens when the start of the new prompt matches the cached prefix byte for byte. How it works The cache reuses a prefix only up to the first point where the new prompt differs. So the order of your prompt parts is everything: put the stable, reusable content first, and the changing content last. If you instead put the changing question at the top, the prefix differs on every call and the cache never

Challenge: The Cacheable Prefix