Token Throughput

Throughput

Part of: Streaming, Latency & Caching

A model with a snappy 200ms time to first token can still take twelve full seconds to answer, because the reply is 1,500 tokens long and the model only emits 130 of them per second. Once the first word lands, the whole rest of the wait is governed by a single number you rarely see on a dashboard: token throughput . What it is Token throughput is the rate at which a model emits output tokens, measured in tokens per second (TPS) . Lesson 4 looked at job-level throughput across many requests; this is the throughput inside one response , how fast a single answer types itself out after generation begins. It is the inverse of the inter-token latency from lesson 1: if each token arrives every 8 milliseconds, that is 1 / 0.008 = 125 tokens per second. The longer the answer, the more this rate dominates the clock. How it works Total response time splits into two parts: the one-time TTFT to produce the first token, and then the steady stream of remaining tokens at the throughput rate. The decisive insight is the mix . For a 10-token reply, TTFT (0.2s) swamps the 0.07s of streaming, so first-token speed is everything. For a 1,500-token reply, TTFT is a rounding error and throughput owns the w

Challenge: The Throughput SLO Gate