Cost, Caching, and Batching at Scale

embedding cost, caching duplicates, batching large corpora

Part of: Semantic Search Engine

Every embedding call costs tokens, and a real corpus doesn't sit still. Documents get added, edited, and re-indexed. This lesson is about not paying to embed the same text twice, and not sending a batch so large the API rejects it. Cache identical texts Corpora repeat text. Two support articles share a boilerplate paragraph, or the same FAQ gets synced from two sources. Re-embedding both spends money and time on an identical result. A cache keyed by the exact text, in memory or in a small database, embeds each unique string once: Two things happen here. to embed filters out anything already cached before any API call, and the loop below chunks whatever's left into fixed-size batches. Rebuild the same corpus a second time with this function and it makes zero API calls, since everything is already cached. Batching for API limits, not just cost Embedding providers cap how many texts (and how many total tokens) one request can carry. A corpus of 10,000 unique documents isn't one call. It's however many batches of, say, 128 documents it takes to cover them: ceil(10000 / 128) = 79 calls. Chunking a list into fixed-size pieces is the same pattern whatever's inside it: Estimating cost befo

Challenge: Dedupe, Batch, and Bill