Map-Reduce for Very Long Docs

map-reduce summarization

Part of: AI Text Summarizer

You chunked a 200-page report into 60 pieces. Now what? You can't just paste 60 chunk summaries back together, that's still a wall of text. The pattern that scales to any length is map-reduce . The two phases - Map: summarize each chunk independently. 60 chunks become 60 short summaries. These calls are independent, so the model sees one chunk at a time and never needs the whole document at once. - Reduce: combine those summaries into one. If they all fit in a single call, summarize the summaries and you're done. If there are still too many, group them, summarize each group, and repeat, a tree that collapses toward a single summary. Read the reduce loop carefully: each pass groups the current summaries (here 5 at a time), summarizes each group down to one, and loops until a single summary remains. Ten summaries become two, then one. A hundred become twenty, then four, then one. Why not just summarize once? Because a single chunk summary already loses detail, and stacking 60 of them re-creates the length problem you started with. The reduce phase compresses again, and this time the model synthesizes: it finds the through-line across chunks instead of gluing fragments end to end. The

Challenge: Count the Map-Reduce Calls