Walking the Repo, Safely

repo ingestion

Part of: Codebase Assistant

Before you can answer a question about a codebase, you have to read the whole thing without choking on the junk. This lesson builds the ingestion step: walk a repository, keep the source files, drop everything that isn't code. What we're building By lesson 8 you'll have a command-line tool. Point it at a repo, ask a question in English, and get back an answer with the exact file and line numbers it came from. Every RAG product starts the same way: get the raw documents into memory. For a codebase the documents are source files, and most of a repo isn't source. The problem: repos are mostly noise A typical repository is full of things you never want to feed a model: .git/ internals, node modules/ with tens of thousands of files, compiled artifacts, images, lockfiles. Read every file under the root and you'll spend your token budget on a package-lock.json before you reach the function that answers the question. How real ingestion works Two details matter. dirnames[:] = ... mutates the list os.walk is iterating, in place. That is the only way to stop it descending into node modules at all, instead of descending in and filtering the results afterward. Filtering by extension up front me

Challenge: Filter the Repo Tree