Alerting and incident response

Open Alerts to move from detection to acknowledgement, escalation, and resolution. A rule watches a query or a figure and opens an incident when its condition holds. The escalation policy of the incident pages the people and the on-call schedules it names, and moves on when nobody acknowledges. Notification channels deliver each page. Silences pause notifications during maintenance, and semantic groups collapse related incidents.

The alert center

Section	Purpose	Guide
Incidents	What fires now: acknowledge, silence, resolve, and investigate.	Incidents
Rules	Threshold, anomaly, and real-time rules.	Alert rules
Status pages	Tell customers about incidents and maintenance.	Status pages
History	Past incidents, searchable by service and state.	History
Insights	Incident volume, recovery time, noisy rules, and affected services.	Insights
Groups	Semantic groups that collapse related incidents.	Semantic groups
Escalations	Who is paged, in which order, and when to move on.	Escalation policies
On-call schedules	Rotations and temporary coverage.	On-call schedules
Silence policies	Pause notifications for matching incidents.	Silence policies

Set up alerting

A workspace usually gets ready for its first page in this order:

Connect a channel

Configure a connector such as email, an IM app, or phone and SMS under Settings → Notify → Connectors, and have members bind their personal methods in Account settings → Notify.

Build the on-call schedule

Create an on-call schedule with the rotation of the team.

Create the default escalation policy

Create a policy named default that pages the schedule first and a lead next. Rules saved in the rule workbench use it. See The default policy.

Create and test rules

Start from a preset in the rule workbench, test it on real data, and save it.

Rehearse

Trigger a rule on purpose with Trigger a rule, and check that the page arrives, the escalation moves on, and acknowledgement stops it.

See Make production alerts actionable and Set up on-call and notifications for advice on what to alert on and how to route it.

Permissions

Permission	Capability
`alerts.read`	Read rules, incidents, history, insights, groups, silences, and notification resources.
`alerts.manage`	Create and change rules, escalation policies, groups, connectors, templates, policies, and fallback routes.
`alerts.acknowledge`	Acknowledge and resolve incidents and groups of incidents.
`alerts.silence`	Create, change, and remove silences.
`schedules.read`	Read schedules and who is on call.
`schedules.manage`	Create and change schedules, rotations, and overrides.

Alert rules

Create, test, and tune rules.

Incidents

Acknowledge, investigate, and resolve.

On-call and escalation

Schedules, overrides, and escalation policies.

Notification channels

Connectors, personal methods, templates, policies, and fallbacks.

Get started

Data

Observe

Query and correlate

Reliability

AI ops

Collaboration

Admin

Deploy & operate

Best practices

Resources

Alerting and incident response

The alert center

Set up alerting

Permissions

Alert rules

Incidents

On-call and escalation

Notification channels

​The alert center

​Set up alerting

​Permissions

Alert rules

Incidents

On-call and escalation

Notification channels

The alert center

Set up alerting

Permissions