alerts.incidents.escalate; assigning roles, managing tasks, and posting updates require alerts.incidents.update; changing the
organization’s major incident settings requires alerts.response.manage; reading postmortems requires
postmortems.read, and writing them requires postmortems.manage.
Declare a major incident
Under Major incident in the incident details, select Declare major incident. Then:- The person declaring becomes the Commander, and the Checklist of the organization’s settings becomes tasks.
- A meeting is set up as the settings say: a Feishu meeting or a Tencent Meeting is booked, a Zoom meeting is started, a Google Meet is created, or the fixed meeting link is used; see Meetings.
- When the settings name a connector for war rooms, it makes a group chat (in Feishu, Slack, DingTalk, or
WeCom) named
[severity] incident summary, adds the incident’s assignees, roles, and stakeholders, and posts an opening message with the incident and meeting links. - Every ticket connector with Make a ticket when a major incident is declared creates a ticket.
Roles
Choose a person for each role on the card, or Not assigned to clear it. Assignments go on the timeline.
War room
War rooms are made with the notify connector of the organization’s settings, one that can make group chats:- Feishu: the app needs to create chats, get a chat’s share link, and send messages as the app. Members
are added by the
open_ideach person linked to that connector. - Slack: the Slack App makes a public channel and needs to create channels, invite members, and post messages. Members are invited by the member ID each person linked.
- DingTalk: the internal app makes an internal group and needs the group management permission
(
qyapi_chat_manage). Members are added by the userId each person linked, and group names are at most 20 characters. Making the group does not add the app’s robot to it: add the robot to the group in DingTalk for the opening message and the updates to be posted there. - WeCom: the self-built app makes an app group chat and has to be visible to the root department. Members are added by the userid each person linked, at least two of them.
Meetings
The SecretKey, the Client Secret, and the service account key are stored encrypted and not shown again; left
empty while editing, they keep their values. When the meeting changes, the old secret is no longer kept.
Tasks
The checklist is added on declaring; after that, Add a task, check tasks off, or delete them on the card. A finished task shows who finished it, and adding and finishing tasks go on the timeline. An incident has at most 50 tasks.Post an update
Select Post an update, fill in Impact, Where things are, and Next steps (at least one), and Send. The update:- goes on the incident’s timeline;
- is posted to the war room;
- notifies the incident’s assignees, roles, and the stakeholders of the organization’s settings, sent by the built-in policy Major incident update: notify the stakeholders.
Major incident settings
Open Alerts → Postmortems and select Major incident settings:Postmortems
A postmortem is the incident’s internal write-up, one per incident, kept apart from the status page’s postmortems. Under Postmortem in the incident details, select Write postmortem, and MoleSignal drafts it from a template: the title is the incident summary, and the body has the sections Summary, Impact (how long, which services, the severity), Timeline (the trigger, the steps on the timeline, and the recovery), Root cause (filled from the AI root cause when there is one), What went well, What went badly, and Action items. Select Open postmortem later to carry on:- Change the Title, the Status (Draft, In review, Published), and the Text (Markdown), then Save.
- Draft with Mole Agent starts a Mole Agent chat that writes the draft from the timeline, notes, root cause analysis, and triggering query, and saves it to this postmortem.
- Copy summary copies the text of the Summary section, ready for the status page’s postmortem.
- Action items: Add an action item, and check it off when done; with an enabled ticket connector, Create ticket makes a ticket for it and shows the key beside it.