Chunk by Function, Not by Character

function-level chunking

Part of: Codebase Assistant

A naive chunker slices code every N characters, and it will cut a function in half. One chunk ends up with a signature and no body; the next has a body with no idea what function it belongs to. Code chunking has to respect syntax, not just length. What we're building This lesson replaces "split every 500 characters" with "split at every function and class." The unit of meaning in code is the function, not the paragraph. Everything inside def load config(path): belongs together, and mixing it with half of the next function makes retrieval worse. Why character-based chunking fails on code Text chunkers built for prose (split on blank lines, split every N tokens) assume meaning lives in nearby sentences. Code doesn't work that way. A function's signature, body, and closing line are one unit whether it runs 3 lines or 300. Cut it at a fixed character count and you'll split the signature from the logic, so neither half makes sense on its own once it's retrieved and shown to the model. How real chunking works Python ships a parser for exactly this. The ast module turns source text into a tree of nodes, and every node knows which lines it spans. node.lineno and node.end lineno are 1-index

Challenge: Find the Function Boundaries