Batching & Concurrency

Throughput

Part of: Streaming, Latency & Caching

You have 1,000 documents to classify. Sent one at a time, each waiting for the last to finish, the job takes an hour. Sent with 50 requests in flight at once, it finishes in a couple of minutes, same model, same total work, completely different wall-clock time. The difference is throughput , and the levers are batching and concurrency . What it is Throughput is how much work you finish per unit of time, requests per second, or tokens per second across a whole job. It is a different question from latency , which is how long a single request takes. A model can have high latency per call yet enormous throughput if it handles many calls in parallel. Two ways to raise throughput: - Concurrency : send many independent requests at the same time so their wait times overlap instead of stacking up. - Batching : hand the model several inputs in one request (or one internal batch) so it processes them together far more efficiently than one by one. How it works Most of a request's latency is spent waiting on the network and the model, not on your CPU. So while one request waits, you can have dozens of others waiting too. Serial code wastes that idle time: With N requests each taking t seconds:

Challenge: The Concurrency Planner