Failures & Fallbacks

Fallbacks

Part of: Deploying & Monitoring LLM Apps

Your app worked perfectly for a month. Then the model provider had a 40-minute outage, and because every request called that one API with no plan B, your entire product showed a spinning wheel and then an error to every single user. The outage was theirs. The total failure was yours, you depended on something that will, eventually, fail, and you had no fallback . What it is A fallback is what your system does when the primary path fails. LLM calls fail in many ways: the provider is down, the request times out, you hit a rate limit, or the model returns malformed output. A production app treats every one of these as expected, not exceptional, and has a degraded-but-working answer ready. The two core tools are retries (try again, because many failures are transient) and fallbacks (switch to a backup when retries won't help). The art is knowing which failure deserves which response. How it works A resilient call layers retry, timeout, and a backup: Notice the decisions. A rate limit gets exponential backoff : wait longer each try, because hammering a throttled API makes it worse. A timeout is retried because it is often transient. A hard error breaks out immediately rather than wastin

Challenge: The Resilient Caller