Decide who answers what
Before configuring anything, write down for each service: the team that owns it, who is on call first, who comes next, and which severities justify a phone call at night.
Group the people of each owning team in a team; schedules, escalation
targets, and fallback routes all refer to teams.
Give every person a verified way to be reached
- An administrator configures the connectors under Settings → Notify → Connectors: an IM app, email, and phone and SMS for critical pages. Send test on each one.
- Each member binds personal methods under Account settings → Notify and sends a test to each.
- An administrator with
org.members.manageVerifies each method after confirming that it belongs to the person. Unverified methods never deliver. - Each member orders the methods for Alert events, with waits that escalate the channel: for example IM at once, SMS after 3 minutes, a phone call after 5 minutes. Later steps stop as soon as someone acknowledges.
Build schedules without gaps
Create one on-call schedule per team, in the timezone of the team:- a Weekly rotation with a handoff in working hours, when both people are awake;
- for follow-the-sun or business-hours coverage, a first rotation restricted to working hours and a second rotation below it for the remaining hours;
- every member, in the order of the rotation, and at least two people per schedule.
Create the default escalation policy first
Rules saved in the rule workbench use the policy nameddefault; without it, their incidents page nobody.
See The default policy.
Turn on Repeat the whole policy until acknowledged with a small Max loops for critical services.
Create further policies for teams with different needs, and point rules at them through the API with
escalation_policy_id.
Add safety nets
- Fallback routes deliver to a team channel or an organization address when no personal method of a recipient works.
- Notify policies decide the template and the connectors of each event; the live preview shows who would receive an event without sending it.
- The Delivery log shows every attempt, so a failing connector is found before an incident needs it.
Rehearse
Run a drill in working hours, and repeat it after every larger change:- Fire a test rule on purpose with Trigger a rule.
- Check that the person on call receives the page on each channel, in the expected order.
- Wait without acknowledging, and check that the escalation reaches step 2.
- Acknowledge, and check that later steps and channels stop.
- Resolve, and review the Delivery log for failures.
Acceptance criteria
- Every person on a schedule has at least two verified methods, one of them by phone for critical pages.
- No schedule shows a coverage gap in the next 7 days.
- A policy named
defaultexists, and each step has a person or a schedule. - Fallback routes exist for every team that receives pages.
- The last drill reached the person on call, escalated as expected, and stopped on acknowledgement.