Chunking: Cutting Documents Down to Size

Chunking

Part of: Build a RAG System

You can't stuff a 50-page PDF into every prompt. It's expensive, it blows past context limits, and burying the one relevant paragraph in 49 pages of noise makes the model's answer worse , not better. So before anything else, you split documents into chunks, small, searchable pieces. Get this step wrong and the rest of your RAG system retrieves garbage. Why not one chunk per document? Because retrieval needs to be precise. If a document is one giant blob, searching it is all-or-nothing: you either dump the whole thing into the prompt or skip it. Chunks let you pull just the passage that answers the question and leave the rest behind. The tension: too big vs. too small - Chunks too big → each one covers many topics, so similarity search gets fuzzy and you waste tokens shipping irrelevant text. - Chunks too small → you slice a sentence away from the context that gives it meaning. "It costs $4.2M" is useless without the sentence naming what "it" is. A common starting point is a few hundred words per chunk. Don't agonize over the exact number, measure retrieval quality and adjust. Overlap saves split-up ideas Fixed-size chunking has an ugly failure: an important fact lands right on a ch

Challenge: The Sentence-Safe Chunker