> ## Documentation Index
> Fetch the complete documentation index at: https://docs.molesignal.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Make production alerts actionable

> Design alerts around customer impact, reduce transient noise, and verify the full path from detection to recovery.

When a service produces repeated pages but few useful interventions, review the detection and response policy together. Start with customer-visible failures and assign a clear action to every paging rule.

## Page on impact

Use failed business requests, sustained latency, or a failed critical-path [synthetic check](https://docs.molesignal.com/en-US/synthetics.md) as starting signals. An [SLO burn-rate alert](https://docs.molesignal.com/en-US/apm#burn-rate-alerts) pages on the error budget instead of on every spike. Route diagnostic symptoms to dashboards and investigation views unless immediate action can prevent customer impact.

For a low-traffic service, pair an error ratio with a minimum request count in the query. One failed request out of one request is a high ratio but weak evidence of a widespread outage. Choose thresholds from representative normal and incident periods rather than copying values from another service; **Test rule** in the rule workbench draws the query over the last 6 hours with the threshold line.

## Match the rule to the question

| Situation | Recommended rule | Attention point |
| - | - | - |
| Sustained error rate or latency | A [threshold rule](https://docs.molesignal.com/en-US/alerting/rules.md#create-a-threshold-rule) with **And for** a few minutes | Longer confirmation reduces transient noise but delays detection. |
| The same failure on many hosts or services | A threshold rule with [**Split by**](https://docs.molesignal.com/en-US/alerting/rules.md#split-a-rule-by-columns) | Each group opens its own incident, so split by stable columns only. |
| A changing baseline with useful history | An [anomaly rule](https://docs.molesignal.com/en-US/alerting/rules.md#anomaly-rules), with **Weekly seasonality** for weekly traffic | Releases, seasonality, and traffic changes can alter the baseline. |
| A burst of matching events requiring immediate action | A [real-time rule](https://docs.molesignal.com/en-US/alerting/rules.md#real-time-rules), created through the API | Counts are local to each intake node and reset after restart. |

The minutes of **And for** become a number of evaluations, so choose them alongside the evaluation interval and the query window. Overlapping windows reuse observations; consecutive matches are not independent samples. Do not apply that debounce design to real-time rules, which only accept `for_periods` values of `0` or `1`.

Set a [recovery threshold](https://docs.molesignal.com/en-US/alerting/rules.md#recovery-and-missing-data) on the healthy side of the trigger, so a value that hovers around the line does not fire and recover over and over. Decide what missing data means: **Zero** suits counts, and **Open a no-data incident** suits a source that must never go quiet.

## Make ownership and escalation explicit

Fill in the **Service** and a **Response guide / Runbook URL** on every rule that pages. Who is paged comes from the [escalation policy](https://docs.molesignal.com/en-US/alerting/on-call.md#escalation-policies), not from the rule; rules saved in the rule workbench use the policy named `default`. Prefer stable split columns and labels. Unique request identifiers are investigation context and should not split each occurrence into a separate incident.

Set an acknowledgement timeout that fits the severity and staffing model. Check **Next 7 days** of each schedule across timezone boundaries and handovers. Test the next escalation step by deliberately leaving a controlled incident unacknowledged.

During planned maintenance, use a narrowly scoped, time-bounded [silence policy](https://docs.molesignal.com/en-US/alerting/silences.md#silence-policies) that names the affected service and ends by itself. A silence suppresses delivery; it does not establish that the service recovered.

## Rehearse the complete response

A successful **Test connection** of a connector only verifies part of the path. Fire a controlled rule with [Trigger a rule](https://docs.molesignal.com/en-US/api/alerting/trigger-rule.md) and verify incident creation, recipient routing, actual receipt, acknowledgement, and recovery. Check that an acknowledged incident stops escalating and that a recovered condition reaches the expected resolved state.

Include a failed primary delivery path in the exercise. Verify the configured [fallback route](https://docs.molesignal.com/en-US/notification-channels#configure-fallback-routes) with the intended recipients, and read the attempts in the [delivery log](https://docs.molesignal.com/en-US/notification-channels#inspect-and-retry-deliveries), instead of assuming that a fallback definition guarantees delivery.

## Review after an incident

Compare detection delay, acknowledgement delay, duplicate pages, and missed customer impact. [Alerts → Insights](https://docs.molesignal.com/en-US/alerting/incidents.md#insights) lists the **Top firing rules** and the share of **Rapid recoveries**, a sign of noisy rules, and the **Weekly alert noise review** template of [automations](https://docs.molesignal.com/en-US/automations.md) has Mole Agent propose how to tune each noisy rule.

Remove rules without a useful response, refine noisy thresholds, and retain evidence explaining each change. Confirm the new policy against both normal traffic and a known failure scenario.

## Acceptance criteria

* Every paging rule starts from customer impact: failed requests, latency, a failed check, or SLO burn.
* Every paging rule names a service and a runbook, and its escalation policy reaches someone.
* Rules recover on the healthy side of the trigger, and their choice for missing data is deliberate.
* A controlled test reaches the on-call person, escalates when nobody acknowledges, and resolves.
* The top firing rules are reviewed every week.

## Related guides

* [Alert rules](https://docs.molesignal.com/en-US/alerting/rules.md)
* [On-call and escalation](https://docs.molesignal.com/en-US/alerting/on-call.md)
* [Set up on-call and notifications](https://docs.molesignal.com/en-US/best-practices/on-call-setup.md)
* [Work an incident from page to postmortem](https://docs.molesignal.com/en-US/best-practices/incident-response.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.