Watching Cost as the FAQ Grows

cost and token budget

Part of: FAQ Support Bot

Context stuffing works fine at 10 FAQ lines and gets expensive fast at 500. Every question resends the entire FAQ, so the cost of this bot scales with how big your docs are, not how big the question is. This lesson builds the budget check that catches an FAQ before it grows past what's sane to stuff. Why cost scales with the doc, not the question Every call to this bot sends the full numbered FAQ in the system prompt and a short question in messages. A one-word question and a ten-word question cost almost the same, because the FAQ dominates the token count. The lever you actually control is how big the FAQ is allowed to get , not how you phrase questions. Estimating the cost per call system template overhead accounts for the fixed wording around the FAQ: the rules, the "You are a support bot" framing, tokens that exist on every call no matter the doc size. Add it in explicitly rather than pretend the template is free. Setting a hard ceiling This check runs once when the FAQ loads, not on every question, since the doc size doesn't change per call. When it fails, that's not a bug to patch. It's a signal you've outgrown context stuffing and need the retrieval step this project left ou

Challenge: Enforce the FAQ Token Ceiling