Why Streaming Matters (Time To First Token)

Streaming

Part of: Streaming, Latency & Caching

A user waits eight seconds staring at a spinner, then a full paragraph appears at once. The same answer, streamed, shows its first word in 400 milliseconds and feels instant. The total time barely changed, but the perceived time collapsed, and that gap is the reason streaming exists. What it is Streaming means sending the model's output to the user token by token as it is generated, instead of waiting for the whole response and shipping it in one block. The model already produces text one token at a time (that autocomplete loop), so streaming just stops hiding that process behind a wall. The number that captures the win is time to first token (TTFT) : how long from sending the request until the very first token shows up. With streaming, TTFT is small. Without it, TTFT equals the total generation time, because nothing appears until the end. How it works A non-streaming call blocks until done, then returns everything. A streaming call opens a connection and pushes each token as the model emits it: Two latency numbers now matter instead of one: 1. TTFT : time until the first token, set mostly by the model processing your prompt before it can speak. 2. Inter-token latency : the gap bet

Challenge: The TTFT Router