Batch and Async Discounts

Batch

Part of: Cost & Token Budgeting

You have a nightly job: summarize 100,000 documents before the morning report. Run them through the normal real-time API and you pay full rate and watch progress bars all evening. Send them to the batch API instead and the same work is billed at half price - the catch is it might take a few hours to come back. For an overnight job, that trade is a steal. What it is The batch API is a discounted lane for work that does not need an instant answer. You submit a big pile of requests at once, the provider processes them whenever it has spare capacity, and you get all the results back within a turnaround window (often up to a day). In exchange for giving up immediacy, you get a flat discount - commonly around 50% off both input and output - on the whole batch. The trade is latency for price . Real-time gets you an answer in seconds at full cost; batch gets you the answer in minutes-to-hours at a fraction of the cost. How it works The decision is per-job: does this task tolerate the batch turnaround? If yes, route it to batch and apply the discount. If a job needs a fast answer (a user is waiting), it stays on the real-time lane at full price: So a job that can wait pays half; a job that

Challenge: The Overnight Router