Multiple Chunks: Top-K and Deduplication

top-k retrieval and deduplication

Part of: Chat With Your PDF

A real question is often answered by more than one paragraph, spread across the document. So you retrieve the top few chunks, not just the single best one. But the overlapping windows from lesson 1 introduce a new problem: several of your top matches can be near-duplicates of each other, and three copies of the same sentence eat prompt budget without adding a thing. Top-k retrieval Top-k means "take the k highest-scoring chunks instead of one." A typical RAG app sets k between 3 and 6. That's enough for a document with no overlap. But your chunker used a sliding window, so chunk 4 and chunk 5 can share most of their text by design. If both score high, top-k hands the model the same sentence twice under two different labels. Deduplicating near-identical chunks Here's a cheap check that works: if one chunk's text sits entirely inside another's, they say the same thing, so drop the shorter one. Combine them: scan until you have k unique chunks Don't dedupe after slicing the top k, because that can leave you with fewer than k chunks. Instead, scan in score order, skip duplicates as you go, and stop once you've collected k unique chunks: This keeps looking past the naive top-k cutoff, s

Challenge: Fill Top-K After Deduping