Where the Money Goes

Cost Drivers

Part of: Cost & Token Budgeting

A team launches a chatbot, watches the first few messages cost almost nothing, and relaxes. Three weeks later the bill is ten times what they modeled. Nothing broke. They just didn't see where the money actually goes in a real conversation, and it's almost never where beginners expect. What it is Two things dominate a real LLM bill, and neither is the obvious "the prompt is long." 1. Long outputs. Output tokens cost several times more than input, so an answer that rambles is the single biggest controllable cost on any one call. 2. Chat history that compounds. This is the sneaky one. A model is stateless, it remembers nothing between calls. To fake a conversation, the app resends the entire transcript every single turn . Turn 10 doesn't just send your latest message; it re-sends turns 1 through 9 as input, plus the new message. How it works That resending makes input cost grow with every turn. If each turn adds about the same number of tokens, the running input total climbs like 1, then 1+2, then 1+2+3, the transcript gets re-billed again and again: By turn 20 of a long chat, you might be paying to resend nineteen earlier messages on every reply. The conversation didn't get smarter;

Challenge: The Stateless Transcript