> ## Documentation Index
> Fetch the complete documentation index at: https://docs.molesignal.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Discovery

> Problems found without a rule: sudden moves of the errors, latency, and traffic of APM services, new versions doing worse, new and returning front-end errors, surges and new patterns of error logs, and hosts and containers in trouble; turned into incidents with a discovery rule, and looked into by Mole Agent, when needed.

Every 5 minutes, discovery goes through each organization and keeps what it finds as **findings**. Open
**Alerts → Discovery**. Reading it takes `alerts.discovery.read`, and a rule takes `alerts.rules.manage`.

## What it looks for

| Kind | From | When |
| - | - | - |
| **Service move** | [APM insights](https://docs.molesignal.com/en-US/apm/insights.md) | In the last half hour, a service's error rate, P95 latency, or traffic moved clearly against both the stretch before and the same stretch a week before, with enough requests to tell. |
| **Worse new version** | APM insights | A version first seen in the last half hour does worse than the version before it. |
| **New front-end error** | [RUM errors](https://docs.molesignal.com/en-US/rum/errors.md) | An error issue an application first had in the last half hour. |
| **Front-end error back** | RUM errors | A resolved error issue seen again in a newer version. |
| **Error log surge** | Logs | In the last 15 minutes, a log stream wrote at least 20 error logs and at least 3 times its usual count per 15 minutes in the hour before; split by service when the stream has a service field. |
| **New error log** | Logs | An error message seen at least 3 times in the last 15 minutes that the stream did not have in the day before. Numbers and hexadecimal strings in a message count as the same message; at most 10 per stream. |
| **Running out of room** | Metrics | A file system at least 90% full, or memory at least 95% used. |
| **CPU jump** | Metrics | A host's CPU at least 60% busy and at least 30 points busier than an hour before. |
| **Container restarts** | Metrics | A container restarted in the last 15 minutes. |

Logs: each organization's 20 log streams with a level field (`level`, `severity`, `severity_text`,
`severity_number`, and the like), the most recently changed first. Error, fatal, critical, and their
kin are errors, as is a `severity_number` of 17 or more. Metrics: a fixed set of common resource metrics
of the OpenTelemetry host and Kubernetes receivers, Prometheus node\_exporter, and kube-state-metrics,
queried only when the metrics directory has them; no other metric is scanned.

Performance suggestions (slow dependencies, N+1 calls, slow statements, hot endpoints) are not
problems and are left out, and so are ignored and resolved issues.

A problem (the same kind, service or application, environment, and metric) has one open finding at a
time: seeing it again updates the figure and when it was last seen; a finding not seen for three rounds
in a row ends, and ended findings are kept for 30 days. When one side cannot be read, for example APM or
RUM cannot be queried for a while, or a log stream fails, the open findings of that side stay as they
were that round. With
several `alert` nodes, organizations are divided among them by id.

## Findings

Switch between **Open** and **Ended**, and filter by kind. Each row says in a sentence what was found,
for example "The error rate of checkout is 12.0% (was 0.40%)", with the severity, the kind, and the
environment below, then when it started and when it was last seen (or when it ended).

Select a row for its details: the kind, the figure now, before, and a week before, the version, where the
error happens, and when it started and was last seen; an APM finding also lists the deployments and
versions near when it began. **Open in APM**, **Open in RUM**, **Open the logs**, and **Open in Metrics**
go to the service, the application, the log stream, or the query.

**Ask Mole Agent to look into it** closes the details and starts a [Mole Agent](https://docs.molesignal.com/en-US/agent) chat about
the finding: Mole Agent reads it with `get_discovery_finding`, looks at the logs, metrics, traces, and
changes around when it began, and says what to do next.

## Be told with a rule

Findings are only listed and wake nobody: they are inferred, and paging on them directly would be noisy.
To be told, create a discovery rule:

1. Select **New discovery rule**, or **Alert on findings like this** in a finding's details, which fills
   in its kind, service, and environment.
2. Under **Which findings**, enter a **Name**, choose the **Kinds** (none ticked is every kind), and enter
   the **Services or applications** (APM service names or RUM application ids, separated by commas;
   empty is all), the **Environments**, and the **Minimum severity**.
3. Under **Who is told**, choose the **Escalation policy** and the **Urgency**: **Low (default)** notifies
   the first step only and never during quiet hours.
4. Select **Create rule**.

Every minute, the rule looks at the open findings: each finding it takes opens its own incident, as
severe as the finding, with the labels `discovery.finding`, `discovery.kind`, and `discovery.subject`
that routing rules and silences can match. When the finding ends, the incident recovers. The rule shows
in the [rule list](https://docs.molesignal.com/en-US/alerting/rules.md#discovery-rules) with **Discovery** as its source.

## Let Mole Agent look into new findings

An [automation](https://docs.molesignal.com/en-US/automations.md#what-starts-it) with the trigger **New discovery findings** runs for each
finding that appears after it was saved or switched on and matches its **Kinds** (none ticked is every
kind), **At least** a severity, and **Services, hosts or containers** (separated by commas; empty is all of
them). A finding is looked into once. Findings start nothing by default, so that inferred problems do
not spend the token budget unasked.

## API and Mole Agent

| Action | Endpoint or tool |
| - | - |
| List findings | [`GET /api/v1/alerts/discovery/findings`](https://docs.molesignal.com/en-US/api/alerting/discovery) |
| One finding | [`GET /api/v1/alerts/discovery/findings/{id}`](https://docs.molesignal.com/en-US/api/alerting/discovery) |
| Create a discovery rule | [Create a rule](https://docs.molesignal.com/en-US/api/alerting/create-rule.md) with `query.discovery` |
| Mole Agent | `list_discovery_findings` and `get_discovery_finding` (read only) |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.