Semantic Caching

SemanticCache

Part of: Streaming, Latency & Caching

Two users ask "How do I reset my password?" and "What's the process for changing my password?" Word for word they share almost nothing, so the exact-match prompt cache from lesson 3 misses both times and you pay for two full model calls. But the meaning is identical. A semantic cache catches that, it matches by what a query means, not by the letters it is made of. What it is A semantic cache stores past queries and their answers keyed by an embedding : a vector of numbers that captures meaning. A new query is embedded too, and if its vector is close enough to a stored one, you return the cached answer instead of calling the model. Prompt caching (lesson 3) needs a byte-for-byte prefix match; semantic caching tolerates completely different wording as long as the intent lines up. How it works The closeness test is cosine similarity : the cosine of the angle between two vectors, ranging from -1 (opposite) to 1 (identical direction). Near 1 means "these mean nearly the same thing." You set a similarity threshold ; at or above it is a hit, below it is a miss. The flow per query: embed it, compare against every cached embedding, take the best similarity. If that best is at or above the t

Challenge: The Semantic Cache Simulator