Page on impact
Use failed business requests, sustained latency, or a failed critical-path synthetic check as starting signals. An SLO burn-rate alert pages on the error budget instead of on every spike. Route diagnostic symptoms to dashboards and investigation views unless immediate action can prevent customer impact. For a low-traffic service, pair an error ratio with a minimum request count in the query. One failed request out of one request is a high ratio but weak evidence of a widespread outage. Choose thresholds from representative normal and incident periods rather than copying values from another service; Test rule in the rule workbench draws the query over the last 6 hours with the threshold line.Match the rule to the question
The minutes of And for become a number of evaluations, so choose them alongside the evaluation interval and the query window. Overlapping windows reuse observations; consecutive matches are not independent samples. Do not apply that debounce design to real-time rules, which only accept
for_periods values of 0 or 1.
Set a recovery threshold on the healthy side of the trigger, so a value that hovers around the line does not fire and recover over and over. Decide what missing data means: Zero suits counts, and Open a no-data incident suits a source that must never go quiet.
Make ownership and escalation explicit
Fill in the Service and a Response guide / Runbook URL on every rule that pages. Who is paged comes from the escalation policy, not from the rule; rules saved in the rule workbench use the policy nameddefault. Prefer stable split columns and labels. Unique request identifiers are investigation context and should not split each occurrence into a separate incident.
Set an acknowledgement timeout that fits the severity and staffing model. Check Next 7 days of each schedule across timezone boundaries and handovers. Test the next escalation step by deliberately leaving a controlled incident unacknowledged.
During planned maintenance, use a narrowly scoped, time-bounded silence policy that names the affected service and ends by itself. A silence suppresses delivery; it does not establish that the service recovered.
Rehearse the complete response
A successful Test connection of a connector only verifies part of the path. Fire a controlled rule with Trigger a rule and verify incident creation, recipient routing, actual receipt, acknowledgement, and recovery. Check that an acknowledged incident stops escalating and that a recovered condition reaches the expected resolved state. Include a failed primary delivery path in the exercise. Verify the configured fallback route with the intended recipients, and read the attempts in the delivery log, instead of assuming that a fallback definition guarantees delivery.Review after an incident
Compare detection delay, acknowledgement delay, duplicate pages, and missed customer impact. Alerts → Insights lists the Top firing rules and the share of Rapid recoveries, a sign of noisy rules, and the Weekly alert noise review template of automations has Mole Agent propose how to tune each noisy rule. Remove rules without a useful response, refine noisy thresholds, and retain evidence explaining each change. Confirm the new policy against both normal traffic and a known failure scenario.Acceptance criteria
- Every paging rule starts from customer impact: failed requests, latency, a failed check, or SLO burn.
- Every paging rule names a service and a runbook, and its escalation policy reaches someone.
- Rules recover on the healthy side of the trigger, and their choice for missing data is deliberate.
- A controlled test reaches the on-call person, escalates when nobody acknowledges, and resolves.
- The top firing rules are reviewed every week.