> ## Documentation Index > Fetch the complete documentation index at: https://docs.molesignal.com/llms.txt > Use this file to discover all available pages before exploring further. # Insights and investigations > Read what moved against the stretch before and the same stretch a week before, the changes near then, and where time could be won back; hand a finding to Mole Agent, see what an incident reaches, and alert on anomalies of APM figures. The **Insights** card of the APM overview and of a service's overview says what is worth a look in the time range: what moved against the stretch before and against the same stretch a week before, the versions that did worse, and where time could be won back. A finding names the changes near when it began, and **Investigate** hands it to Mole Agent. ## Baselines Above the findings, two lines compare the services in view with two stretches of the same length: * **Against the stretch before**: the stretch that ends where the range begins; * **Against the same stretch a week before**: the range moved back seven days. Each line says how throughput and the error rate moved, the error rate in percentage points. A stretch without requests has no line. Minute aggregates are kept for 24 hours. A stretch that begins earlier is read from the hourly aggregates that cover it, and its requests are counted for the stretch's own length. ## What insights find | Insight | Severity | When | | - | - | - | | **Moved** | **Critical** while the service is critical in the range, else **Warning** | The error rate, P95, or throughput moved against each baseline stretch that has enough requests to judge by, and at least one has them. | | **Regressed** | **Warning** | A version first seen in the range did worse than the version before it, by the verdict of [Deployments](https://docs.molesignal.com/en-US/apm/deployments.md). | | **Bottleneck** | **Suggestion**; **Warning** while the service is not healthy | One dependency takes at least 40% of the time of the service's requests, over at least 20 calls. | | **N+1** | **Suggestion** | The same call, other than to a message broker, is made at least 10 times a request and at least 200 times in all: a loop that one call could replace. | | **Slow SQL** | **Suggestion** | A database statement has a P95 of at least 100 ms over at least 20 calls. The two statements of a service that take the most time in all are named. | | **Hot endpoint** | **Suggestion** | One endpoint takes at least 40% of the service's time, over at least 50 requests, while two or more endpoints served. | A move is judged on each figure on its own: | Figure | Moved when | | - | - | | Error rate | It rose by at least two percentage points, and to at least twice what it was. | | P95 | It rose to at least 1.5 times what it was, and by at least 100 ms. | | Throughput | It fell below half, or rose to at least twice, with at least 100 requests in the busier stretch. | A stretch needs at least 20 requests for its error rate or P95 to count. The finding gives the figure now beside the figure of each baseline stretch. The findings come critical first, then warnings, then suggestions, at most 30. Suggestions say where time goes; they do not say that the service is unhealthy. ### When a move began A move began at the first bucket of the range that stands past the line halfway between the baseline and the figure now, when the next bucket stands past it too and at least half of the buckets after it do. A single bad bucket does not start a move. The card gives the time a move began next to the finding; the exact time shows when pointed at. ## Changes near a finding A moved or regressed finding lists the changes from 30 minutes before it began until 10 minutes after, the nearest first, at most five: * the deployments, configuration changes, and feature flags that pipelines reported for the service, or for a service it calls; * the versions that the service's spans first carried then, which no pipeline reported, marked as first seen in spans. A change reported for an environment counts only in that environment. Report configuration changes and feature flags as deployments with a `kind`; see [Report configuration changes and feature flags](https://docs.molesignal.com/en-US/apm/deployments.md#report-configuration-changes-and-feature-flags). ## Where insights show | Page | What the card holds | | - | - | | **Overview** | The findings of every service in view, each with its service. **Regressed** is left out: the recent deployments card names the versions that did worse. | | A service's **Overview** | The findings of the service. **Bottleneck** and **Hot endpoint** are left out: the time spent downstream and the endpoints that weigh most are on the same page. | When nothing stands out, the card says so. The environment and version filters of the page apply. ## Investigate with Mole Agent **Investigate** opens Mole Agent with a new chat about what you are reading: | Where | The question names | | - | - | | The header of a service page | The service, its environment, and the time range. | | An endpoint drawer | The endpoint, its service and environment, and the time range. | | An error drawer | The error, its fingerprint, its service and environment. | | A finding on the **Insights** card | The finding as the card says it, its service and environment, and the time range. | The service is attached to the chat, and the question waits in the message box to be read, changed, and sent. The button shows to those who hold `agent.use`. Starting the chat needs `streams.query`, or `sys.telemetry.read` in the `_sys` scope, and records an investigation of the service; see [Investigations](https://docs.molesignal.com/en-US/agent#investigations). An attached service gives the model, for the chat's time range or the last hour when it has none: * each environment's health with its reasons, requests, error rate, P50, P95, P99, Apdex, and version; * the figures of the stretch before and of the same stretch a week before; * up to eight findings of the **Insights** card, with the changes near them; * up to six services that call it, and up to six things it calls, with their health. A service without APM data gives the calls between services that the service graph paired over the last day instead. Mole Agent can also read insights with `get_apm_insights` and what a service reaches with `get_apm_blast_radius`. The built-in incident triage skill uses both, and the latency regression skill reads the insights of the service first. ## The blast radius of an incident When an incident names a service, its detail has a **Blast radius** section. The service comes from the labels `service.name`, `service_name`, `service`, or `svc`, else from the incident's affected services. The environment comes from the label `deployment.environment` or `environment`. The section reads the APM service map from 30 minutes before the incident opened until it resolved, or until now while it is open: | List | What it holds | | - | - | | **Calls it: what its trouble reaches** | The services that call the service, and those that call them, up to three calls away. | | **It calls: where the trouble may come from** | The services, databases, caches, external endpoints, and topics that the service calls or publishes to, and what those services call, up to two calls away. | Each list puts the unhealthy first, at most six, each with how far away it is, its health, and the calls between it and the node it is reached through with the share that failed. A topic or an external endpoint is not followed further, and a call through a topic does not reach its producer, since a producer does not wait on its consumers. Calls that are gone against the stretch before are left out. **Open on the service map** opens the [APM service map](https://docs.molesignal.com/en-US/apm/service-map.md#the-apm-service-map) of the environment with the service chosen. The section shows to those who can read APM data, once the map is read. When the service is not on the map in that time, the section says so. ## Alert on anomalies of APM figures An anomaly rule can watch an APM figure instead of the result of a query. It fires when the figure over the rule's window strays from the same window on each of the days before. **1. Open the anomaly rule editor** On a service page or in an endpoint drawer, open **New alert** and choose **Anomaly alert**. The editor opens with the **APM figure** data source and the service, environment, and endpoint filled in. A new anomaly rule in **Alerts** offers the same data source. **2. Choose the figure and the baseline** Choose the figure, then the lookback of 1 to 30 days, the algorithm, and the sensitivity. With **Weekly seasonality**, only the same weekday counts. **3. Save** Name the rule and create it. Creating a rule requires `alerts.manage`. An anomaly rule takes every APM figure except **New error groups** and **SLO burn rate**. A past window older than 24 hours is read from the hourly aggregates that cover it, with requests and errors counted for the window's own length. The rule carries the same labels as a threshold rule on the figure; see [Alert on APM figures](https://docs.molesignal.com/en-US/apm#alert-on-apm-figures). For the algorithms and their parameters, see [Alerting](https://docs.molesignal.com/en-US/alerting). This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.