Fit the Examples in the Budget

selecting few-shot examples under a token budget

Part of: Smart Reply Generator

Few-shot examples make replies sound like you, but every example is text you resend and pay for on every call. As your library of past replies grows, you can't cram them all in. The hardening question is which examples you keep and how many fit. This lesson bounds your cost without wrecking your tone. Tokens are money, examples are tokens Each example (an incoming message plus your reply) costs tokens, and you pay for the whole prompt every call. Left unchecked, a user with 200 saved replies would send a small novel on every draft. So you set a token budget for examples and select the best-fitting subset. Estimate tokens the usual rough way, about 4 characters per token: Newest-first is a good default Which examples to keep? Recency is a cheap, reliable heuristic: your most recent replies best reflect how you write now. Walk your examples newest-first and greedily include each while it still fits the budget. Stop at the first one that would overflow. Two knobs guard cost: a token budget (a hard ceiling on size) and a max count (few-shot rarely improves past 3 to 5 examples, and each one still adds latency). Whichever limit you hit first stops the selection. Why bound it two ways A

Challenge: Budget the Examples