Turn a PDF Into Text You Can Search

chunking a document

Part of: Chat With Your PDF

A chatbot that "reads" your PDF never touches the whole file when you ask a question. It never sees the PDF at all. What it sees are a few small pieces of text you pick out and paste into the prompt. This lesson builds the first piece: turn a document into text, then slice that text into pieces small enough to search and cheap enough to send. What we're building By lesson 8 you'll have a working app. Upload a document, ask a question, get an answer sourced from the document. That is RAG , retrieval-augmented generation: pull the relevant pieces of a document, then generate an answer grounded in them. Every RAG app begins by getting clean text out of the source file. Extracting the text A PDF is a page-layout format, not plain text, so you need a library to pull the text out: The or "" matters: a scanned image page has no extractable text and returns None, which would crash a join. Guard it and move on. Why you can't just paste the whole thing in Two reasons. First, the context window : the model reads only so many tokens per call, and a real document blows past that limit fast. Second, and this holds even when the document does fit: dumping 40 pages into every question burns tokens

Challenge: Chunk With Overlap