Skip to main content
An incident is stressful because several things must happen at once: someone must own it, customers must hear about it, and the cause must be found. Keep one thread: the incident in Alerts. Everything else, from the status page to the postmortem, links back to it.

​
Take ownership

Open the incident from the page, or Alerts → Incidents and the Waiting tab. Select Acknowledge within the ack timeout. Acknowledging stops the escalation and shows the owner to everyone who opens the incident, so nobody else starts the same work in parallel. When several incidents fire for one cause, acknowledge the one that describes the cause, and silence the others while the work goes on. Incidents collapsed by a semantic group can be acknowledged together through the API.

​
Size the impact

Read the incident details before changing anything: Check the SLOs of the affected services: a high burn rate means customers feel it now.

​
Tell customers early

When customers are affected, publish a status page incident before the cause is known:
  1. Create the incident as a draft, select the affected components and their impact.
  2. Publish a first update in the Investigating state: what customers see, and when the next update comes.
  3. Add an update at each change: Identified, Monitoring, Resolved. Published updates are never edited, only followed by new ones.
Publication automation can draft the status page incident from the alert, waiting for approval, so the first update is ready in seconds.

​
Find the cause

Work from the symptom toward the cause, keeping the time window of the incident:
  • Open triggering query replays the query that fired, and Related logs and Related metrics open the context around it. See Cross-signal correlation for the jumps between logs, traces, metrics, and sessions.
  • Look for recent changes: the deployment timeline of the service, and the changes listed near an APM finding.
  • Read the AI root cause, or select Generate. Treat it as a lead to verify, and rate it Helpful or Not helpful.
  • Ask Mole Agent with Investigate on the service page, which attaches the service health, its neighbors, and the findings to the chat.
Write down each hypothesis and its evidence in the incident channel of the team as the investigation goes on. The timeline is needed for the postmortem.

​
Mitigate, then recover

Prefer the fastest safe mitigation, such as a rollback or a feature flag, over a fix under pressure. After mitigating:
  • Watch the triggering query return to normal. An open incident resolves by itself when the rule meets its recovery condition; an acknowledged incident waits for someone to select Resolve.
  • Silence related incidents that only repeat the known problem, with Silence one incident, not the rule itself.
  • Move the status page to Monitoring, then Resolved once the service has stayed healthy.

​
Learn from it

Write the postmortem on the status page incident within days: what happened, the impact, the timeline, the cause, and the changes that follow. Save it as a draft while reviewing, and publish it once the incident is resolved. Mole Agent can draft it from the record of the incident. Then improve detection with Insights: Rapid recoveries and Top firing rules point at noisy rules, and a late page points at a missing or slow rule. See Make production alerts actionable.

​
Acceptance criteria

  • Every incident had an owner within the ack timeout.
  • Customers heard about customer-facing incidents from the status page before asking.
  • The postmortem names the cause, the impact, the timeline, and owners for each follow-up.
  • Each follow-up for detection is a concrete rule change, tested on real data.
Last modified on October 26, 2026