Alerting & Monitoring
Practice this topic in a realistic system design interview
Alerting can go wrong in two opposite ways.
The first is having too few alerts. Suppose the payment service starts failing on a Saturday night. No alert fires. The team finds out on Monday morning from a pile of customer complaints and a drop in revenue.
The second is having too many alerts. Suppose the on-call engineer gets 40 alerts every night. Most of them are about brief CPU spikes or disks at 70% that resolve on their own. After a few weeks, the engineer starts ignoring them. Then one night, a real outage alert arrives, and it gets ignored too.
A good alerting system sits between these two extremes. It notifies people quickly when something important breaks, and stays quiet the rest of the time.
In this chapter, we will learn how monitoring and alerting work together, what makes an alert worth sending, how to set thresholds and use SLOs to decide when to alert, and how to keep alerts from overwhelming the people who respond to them.