> ## Documentation Index
> Fetch the complete documentation index at: https://docs.molesignal.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Major incidents and postmortems

> Declare a high-impact incident major: assign roles, open a war room and a meeting, split the work with a checklist, and post updates in one stream; afterwards write an internal postmortem and track its action items.

An incident with wide impact that several people work on can be declared major: it then has three roles,
a war room and meeting, a task list, and one stream of updates. Once it is over, a postmortem is drafted
from a template, and its action items each get an owner and a due date and can each become a ticket.

Declaring requires `alerts.incidents.escalate`; assigning roles, managing tasks, and posting updates require `alerts.incidents.update`; changing the
organization's major incident settings requires `alerts.response.manage`; reading postmortems requires
`postmortems.read`, and writing them requires `postmortems.manage`.

## Declare a major incident

Under **Major incident** in the incident details, select **Declare major incident**. Then:

1. The person declaring becomes the **Commander**, and the **Checklist** of the organization's settings
   becomes tasks.
2. A meeting is set up as the settings say: a Feishu meeting or a Tencent Meeting is booked, a Zoom meeting
   is started, a Google Meet is created, or the fixed meeting link is used; see [Meetings](#meetings).
3. When the settings name a connector for war rooms, it makes a group chat (in Feishu, Slack, DingTalk, or
   WeCom) named `[severity] incident summary`, adds the incident's assignees, roles, and stakeholders, and
   posts an opening message with the incident and meeting links.
4. Every [ticket connector](https://docs.molesignal.com/en-US/alerting/tickets.md) with **Make a ticket when a major incident is declared**
   creates a ticket.

A war room, meeting, or ticket that cannot be made does not stop the declaration. The card then shows who
declared it, the three roles, the **Join the war room** and **Join the meeting** links (DingTalk and WeCom
groups have no join link and show **War room made** instead), and the tasks. A
resolved incident cannot be declared; when the organization has [declaring on its own](#major-incident-settings)
on, a matching new incident is major from the start.

### Roles

| Role | Responsible for |
| - | - |
| **Commander** | Running the response and deciding what comes next. |
| **Communications lead** | Telling people inside and outside how it is going, for example by [posting updates](#post-an-update) and updating the status page. |
| **Scribe** | Keeping the timeline and the decisions, for the postmortem later. |

Choose a person for each role on the card, or **Not assigned** to clear it. Assignments go on the timeline.

### War room

War rooms are made with the [notify connector](https://docs.molesignal.com/en-US/notification-channels) of the organization's settings,
one that can make group chats:

* **Feishu**: the app needs to create chats, get a chat's share link, and send messages as the app. Members
  are added by the `open_id` each person linked to that connector.
* **Slack**: the Slack App makes a public channel and needs to create channels, invite members, and post
  messages. Members are invited by the member ID each person linked.
* **DingTalk**: the internal app makes an internal group and needs the group management permission
  (`qyapi_chat_manage`). Members are added by the userId each person linked, and group names are at most 20
  characters. Making the group does not add the app's robot to it: add the robot to the group in DingTalk
  for the opening message and the updates to be posted there.
* **WeCom**: the self-built app makes an app group chat and has to be visible to the root department.
  Members are added by the userid each person linked, at least two of them.

In DingTalk and WeCom the commander owns the group, or the first member when the commander has no linked
identity. People who have not [linked an identity](https://docs.molesignal.com/en-US/notification-channels#bind-user-notification-methods)
on the connector are not added: in Feishu and Slack they can join through the join link, while DingTalk and
WeCom groups have none. Teams and Google Chat apps cannot make group chats with the credentials their
connectors hold, so they are not offered.

### Meetings

| Meeting | What happens | What it needs |
| - | - | - |
| **Book a Feishu meeting** | A 6-hour meeting is booked with the commander as its owner. | A Feishu App connector (the war room connector by default) with permission to reserve video meetings. |
| **Book a Tencent Meeting** | A 6-hour meeting is booked under the creator. | A self-built app created in the Tencent Meeting enterprise console: its AppId, SdkId (only if it has one), SecretId, SecretKey, and the creator's userid. |
| **Start a Zoom meeting** | An instant meeting is started with the user in the settings as host. | A Server-to-Server OAuth app in the Zoom App Marketplace with permission to create meetings (`meeting:write:meeting:admin`): its Account ID, Client ID, and Client Secret, and the host's email or user ID. |
| **Create a Google Meet** | A service account creates a meeting for the organizer. | The service account's key (JSON) and the organizer's email; in the Google Workspace admin console, give the service account domain-wide delegation for `https://www.googleapis.com/auth/meetings.space.created`. |
| **A fixed meeting link** | The same link every time. | The meeting link, starting with `https://`. |

The SecretKey, the Client Secret, and the service account key are stored encrypted and not shown again; left
empty while editing, they keep their values. When the meeting changes, the old secret is no longer kept.

### Tasks

The checklist is added on declaring; after that, **Add a task**, check tasks off, or delete them on the
card. A finished task shows who finished it, and adding and finishing tasks go on the timeline. An incident
has at most 50 tasks.

### Post an update

Select **Post an update**, fill in **Impact**, **Where things are**, and **Next steps** (at least one), and
**Send**. The update:

* goes on the incident's timeline;
* is posted to the war room;
* notifies the incident's assignees, roles, and the stakeholders of the organization's settings, sent by the
  built-in policy [Major incident update: notify the stakeholders](https://docs.molesignal.com/en-US/notification-channels#built-in-policies).

## Major incident settings

Open **Alerts → Postmortems** and select **Major incident settings**:

| Part | Settings |
| - | - |
| **War room and meeting** | **Connector for war rooms**: a notify connector that can make group chats (Feishu App, Slack App, DingTalk, WeCom), or **No war room**; see [War room](#war-room). **Meeting**: **No meeting**, or one of the [meetings](#meetings). |
| **Checklist** | Tasks added on declaring, one per line, at most 30. |
| **Stakeholders** | Updates notify these people, and war rooms add them. |
| **Declaring on its own** | With **Declare automatically** on, a new incident at or above the chosen severity, at or above the chosen **Priority** (for example P2 or higher; once chosen, an incident without a priority never matches, and **Any priority** takes all), and whose service is among the given services (empty means any) is major from the start, and the card says it was declared by the organization's rule. |

## Postmortems

A postmortem is the incident's internal write-up, one per incident, kept apart from the
[status page's postmortems](https://docs.molesignal.com/en-US/status-pages#postmortems).

Under **Postmortem** in the incident details, select **Write postmortem**, and MoleSignal drafts it from a
template: the title is the incident summary, and the body has the sections **Summary**, **Impact** (how long,
which services, the severity), **Timeline** (the trigger, the steps on the timeline, and the recovery),
**Root cause** (filled from the AI root cause when there is one), **What went well**, **What went badly**,
and **Action items**. Select **Open postmortem** later to carry on:

* Change the **Title**, the **Status** (**Draft**, **In review**, **Published**), and the
  **Text (Markdown)**, then **Save**.
* **Draft with Mole Agent** starts a Mole Agent chat that writes the draft from the timeline, notes, root
  cause analysis, and triggering query, and saves it to this postmortem.
* **Copy summary** copies the text of the **Summary** section, ready for the status page's postmortem.
* **Action items**: **Add an action item**, and check it off when done; with an enabled ticket connector,
  **Create ticket** makes a ticket for it and shows the key beside it.

**Alerts → Postmortems** lists the organization's postmortems: the status, the owner, how many action items
are open and how many are overdue, and when each was last changed. Select a title to open the postmortem, or
the incident name under it to go to the incident.

## API

| Operation | Endpoint |
| - | - |
| Declare and respond | [`POST /api/v1/alerts/incidents/{id}/declare`](https://docs.molesignal.com/en-US/api/alerting/major-incidents) |
| Post an update | [`POST /api/v1/alerts/incidents/{id}/status-updates`](https://docs.molesignal.com/en-US/api/alerting/major-incidents#post-an-update) |
| Major incident settings | [`PUT /api/v1/alerts/response/settings`](https://docs.molesignal.com/en-US/api/alerting/major-incidents#organization-settings) |
| Postmortems and action items | [`/api/v1/alerts/postmortems`](https://docs.molesignal.com/en-US/api/alerting/postmortems) |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.