A production incident does not usually mean the team failed. More often, the system met a real-world condition it was not prepared to handle.

AI incidents are often harder to notice than typical service failures. The application may stay online while answer quality drops, retrieval data goes stale, a model starts refusing valid requests, or a prompt change breaks one important customer segment. The first report may come from support, sales, trust and safety, or a single customer with a screenshot.

The operating principle is simple: reduce user exposure first, preserve evidence, then investigate. Do not spend the first twenty minutes debating whether the model is wrong while bad outputs continue reaching users.

This chapter covers the incident response practices that AI systems need: runbooks, feature flags, rollback paths, sampling workflows, and post-incident reviews.