Cost and Scale: Deduping Notes Before You Pay to Embed

Drop exact-duplicate chunks before embedding and estimate the token bill, so a template-heavy vault does not triple your index cost.

Part of: Personal Notes Brain

A real notes vault is messy. Daily journals copy-paste the same template into dozens of entries: "Mood:", "Top priority:", the same boilerplate headers. Every repeated chunk costs an embedding call, takes a slot in your index, and crowds genuinely unique content out of retrieval results. Before you scale to hundreds of notes, dedupe. Exact-duplicate detection The cheapest, most reliable dedup catches chunks that are the same content with different casing or spacing, like "Buy milk" against "BUY MILK" against "Buy milk", by normalizing before comparing. Keep the first occurrence's original text so display casing looks natural, and use the normalized key only for comparison. Near-duplicate detection Exact-text dedup will not catch two chunks that say almost the same thing in different words. That needs a comparison of their embeddings , not their raw text. If two chunks' cosine similarity is above a very high bar, say 0.98, they are near-duplicates even with different wording. That check runs after embedding, as a housekeeping pass over the index, using the same cosine similarity function you already built. Budgeting the embedding cost before you index Embedding is billed per token,

Challenge: Dedup Before You Pay to Embed