Insights and investigations

The Insights card of the APM overview and of a service’s overview says what is worth a look in the time range: what moved against the stretch before and against the same stretch a week before, the versions that did worse, and where time could be won back. A finding names the changes near when it began, and Investigate hands it to Mole Agent.

​

Baselines

Above the findings, two lines compare the services in view with two stretches of the same length:

• Against the stretch before: the stretch that ends where the range begins;
• Against the same stretch a week before: the range moved back seven days.

Each line says how throughput and the error rate moved, the error rate in percentage points. A stretch without requests has no line. Minute aggregates are kept for 24 hours. A stretch that begins earlier is read from the hourly aggregates that cover it, and its requests are counted for the stretch’s own length.

​

What insights find

A move is judged on each figure on its own: A stretch needs at least 20 requests for its error rate or P95 to count. The finding gives the figure now beside the figure of each baseline stretch. The findings come critical first, then warnings, then suggestions, at most 30. Suggestions say where time goes; they do not say that the service is unhealthy.

​

When a move began

A move began at the first bucket of the range that stands past the line halfway between the baseline and the figure now, when the next bucket stands past it too and at least half of the buckets after it do. A single bad bucket does not start a move. The card gives the time a move began next to the finding; the exact time shows when pointed at.

​

Changes near a finding

A moved or regressed finding lists the changes from 30 minutes before it began until 10 minutes after, the nearest first, at most five:

• the deployments, configuration changes, and feature flags that pipelines reported for the service, or for a service it calls;
• the versions that the service’s spans first carried then, which no pipeline reported, marked as first seen in spans.

A change reported for an environment counts only in that environment. Report configuration changes and feature flags as deployments with a kind; see Report configuration changes and feature flags.

​

Where insights show

When nothing stands out, the card says so. The environment and version filters of the page apply.

​

Investigate with Mole Agent

Investigate opens Mole Agent with a new chat about what you are reading: The service is attached to the chat, and the question waits in the message box to be read, changed, and sent. The button shows to those who hold agent.use. Starting the chat needs streams.query, or sys.telemetry.read in the _sys scope, and records an investigation of the service; see Investigations. An attached service gives the model, for the chat’s time range or the last hour when it has none:

• each environment’s health with its reasons, requests, error rate, P50, P95, P99, Apdex, and version;
• the figures of the stretch before and of the same stretch a week before;
• up to eight findings of the Insights card, with the changes near them;
• up to six services that call it, and up to six things it calls, with their health.

A service without APM data gives the calls between services that the service graph paired over the last day instead. Mole Agent can also read insights with get_apm_insights and what a service reaches with get_apm_blast_radius. The built-in incident triage skill uses both, and the latency regression skill reads the insights of the service first.

​

The blast radius of an incident

When an incident names a service, its detail has a Blast radius section. The service comes from the labels service.name, service_name, service, or svc, else from the incident’s affected services. The environment comes from the label deployment.environment or environment. The section reads the APM service map from 30 minutes before the incident opened until it resolved, or until now while it is open: Each list puts the unhealthy first, at most six, each with how far away it is, its health, and the calls between it and the node it is reached through with the share that failed. A topic or an external endpoint is not followed further, and a call through a topic does not reach its producer, since a producer does not wait on its consumers. Calls that are gone against the stretch before are left out. Open on the service map opens the APM service map of the environment with the service chosen. The section shows to those who can read APM data, once the map is read. When the service is not on the map in that time, the section says so.

​

Alert on anomalies of APM figures

An anomaly rule can watch an APM figure instead of the result of a query. It fires when the figure over the rule’s window strays from the same window on each of the days before. An anomaly rule takes every APM figure except New error groups and SLO burn rate. A past window older than 24 hours is read from the hourly aggregates that cover it, with requests and errors counted for the window’s own length. The rule carries the same labels as a threshold rule on the figure; see Alert on APM figures. For the algorithms and their parameters, see Alerting.