3:12 a.m., PagerDuty: "Disk 75% staging-db". On-call snoozes. 3:28: "Memory high prometheus-dev". Snooze. 4:05: "Payment error rate 15% prod" — drowned in noise, read at 9 a.m. Estimated loss: four hours of sales.
Alert fatigue is not a tooling problem. It is a contract problem: too many signals without severity, runbook, owner — until the channel becomes spam and real urgency goes unnoticed. Prometheus, Datadog or Grafana do not create noise: absent governance lets it accumulate.
Severity, routing and paging rule
| Level | Channel | Example |
|---|---|---|
| P1 critical | Phone pager | Payment down, data loss |
| P2 | Slack @oncall | p95 2× 30 min |
| P3 | Ticket | 7-day disk trend |
| Info | Dashboard | Capacity planning |
Alert without runbook link and service owner should be forbidden in production. If on-call does not know what to do in sixty seconds, it is not an alert — it is a notification.
Each night page must change a concrete action — otherwise it should never have fired.
User symptoms vs vanity metrics
Good: 5xx rate on checkout; multi-window SLO burn rate. Bad: CPU above sixty percent without correlation; cert expires in thirty days (ticket enough).
Prometheus Alertmanager supports group_by and inhibition: if datacenter is down, inhibiting child alerts avoids useless avalanche. But tooling does not replace signal choice — see Prometheus metrics for instrumenting what matters.
Windows, hysteresis and flapping
for: 10m avoids single spike. Thresholds with margin: re-page only if worse than yesterday and above SLO. Flapping signals bad threshold or bad metric — fix root, do not repeat silencing.
Alert oscillating between fired and resolved every ten minutes wears the team more than real outage. Measure false positive rate quarterly.
Ownership and quarterly review
Audit: fired last ninety days? Action taken? False positive rate? Ruthlessly delete noise. Staging must not page same prod rotation — label env=staging → ticket only.
Each alert needs named owner — not "infra team". Without ownership, nobody defends noise deletion or runbook improvement.
Human on-call: sleep and rotation
Rotation of at least three people, documented handoff, time compensation. Post-incident: missing alert? too noisy? adjust — blameless. Exhausted on-call misses real urgency as much as badly calibrated alert. Also document who can silence an alert and for how long — permanent silence on staging is healthy; permanent silence on payment is untreated outage.
The climax: monitoring that cries wolf
Decide and move forward without blind spots
First reclassify existing alerts by real severity and user impact. Then replace CPU thresholds with user SLO alerts when correlation allows. Add runbook to every alert that wakes someone at night. Schedule quarterly review with explicit noise deletion. Isolate staging from production night rotation. Compare managed monitoring via directory and compare tool. Target over eighty percent actionable night pages.
Frequently asked questions
How many pager alerts in prod?
Few — each page requires immediate action. Rest goes to ticket or dashboard. Under five waking pages per week if well calibrated; beyond that, team normalises noise.
Warning vs critical?
Critical pages with runbook and owner; warning business hours only. Never mix both night channels — destroys on-call credibility.
How to test an alert?
Game day, error injection, verify fire and runbook. Never tested alert is probably false or ignored. Plan quarterly game day.
Infra vs user SLO?
User symptoms first; CPU only if correlation proven. High CPU without user impact should not wake anyone.
An alert that does not change a night action should not wake anyone.