Incident Response for LLM Apps

Incidents

Part of: Deploying & Monitoring LLM Apps

At 2:14 a.m. the error rate spiked. A prompt change shipped that afternoon had slowly poisoned a fraction of answers, and by night a third of requests were failing. The on-call engineer was paged, stared at a dashboard with no playbook, and spent forty frantic minutes guessing before someone said the words that fixed it: "just roll back the last deploy." It worked in ninety seconds. The outage was not the bug. The outage was the lack of a plan for the bug. What it is Incident response is the prepared, repeatable process for when your LLM app misbehaves in production: detect it, mitigate it fast, then learn from it. It rests on three pillars. Rollback is the fastest mitigation, return to the last known-good version (the versioning from lesson 6 is what makes this possible). On-call is the human who is paged and the playbook they follow. The postmortem is the blameless write-up afterward that turns one painful night into a permanent fix. The guiding instinct: during an incident, stop the bleeding first, understand it later. Roll back now; root-cause in daylight. How it works Detection comes from the metrics you already trace (lessons 2 and 4): when a signal like error rate crosses a

Challenge: The Incident Timeline