APM does not require another SDK or telemetry stream. Send well-instrumented traces, and
MoleSignal derives APM data after trace deduplication and before tail sampling. A trace that is
later sampled out can still contribute to APM metrics.
Pages
APM opens on the overview. Each service has a page of its own.
A service page has these sections:
The line after the service name says what the service is, and follows the environment and version
in view:
An archived service also reads Archived there. Favorite at the end of the title row keeps the
service among the reader’s favorites.
The end of the navigation card links the service’s logs, metrics, and profiles. Each link is lit when
the signal carries the service over the day before the end of the range in view, and MoleSignal looks
again every five minutes:
A signal without data is dimmed and opens the Datasource page that connects it. A look that fails
or takes longer than three seconds leaves the link neither lit nor dimmed.
Prerequisites
- OpenTelemetry traces sent to a MoleSignal trace intake endpoint;
streams.queryin an organization workspace; orsys.telemetry.readfor protected_systelemetry.
Instrument services for APM
1
Set stable service resource attributes
Set The exact setup depends on the selected OpenTelemetry SDK or Collector.
service.name on every service. Add namespace, environment, version, and instance
attributes for service filtering or deployment comparison.2
Use the correct span kinds
Create
SERVER spans for inbound requests and CONSUMER spans for consumed messages. Create
CLIENT or PRODUCER spans for downstream calls. MoleSignal uses these span kinds to separate
transactions from dependencies.3
Record status and exceptions
Set OpenTelemetry span status, HTTP or RPC status attributes, and exception events. MoleSignal
uses the recorded status data to classify successful and failed work and build backend error groups.
4
Send traces and open APM
Send OTLP traces, then open APM → Services. Keep the global time range over recent traffic
while the projector flushes the first buckets.
Recommended resource attributes
The Kubernetes attributes and
cloud.region are usually added by the OpenTelemetry Collector’s
k8sattributes and resourcedetection processors rather than by the SDK.
Use low-cardinality values. Do not put request IDs, raw URLs, user IDs, or other unbounded values in
service, environment, version, route, operation, or dependency attributes.
How MoleSignal derives APM data
Services and endpoints
SERVER and CONSUMER spans contribute service and endpoint RED metrics. The API calls an endpoint a transaction. A parentless span
without a kind can contribute to service totals as a compatibility fallback, but explicit span
kinds produce more complete pages.
MoleSignal builds bounded endpoint names from semantic attributes:
- HTTP method plus
http.route, such asPOST /checkout; - RPC service plus method;
- messaging operation plus destination;
- a safe, low-cardinality span name when no stronger semantic identity exists.
/orders/{id} over raw paths such as /orders/83921.
Dependencies
CLIENT and PRODUCER spans contribute dependency RED metrics. MoleSignal classifies targets as a
service, database, cache, messaging system, external HTTP endpoint, external RPC endpoint, or other
dependency. Standard peer.service, db.*, messaging.*, rpc.*, server.*, and HTTP semantic
attributes improve dependency identity.
A SQL database call’s operation is its statement as a template: db.query.summary, else
db.query.text or db.statement, with every literal replaced by ?, comments removed, lists of values
reduced to one ?, and at most 512 bytes kept. Text in double quotes is kept only for databases where
it names a table or a column, such as PostgreSQL and Oracle; for the others it is a value. A call
without a statement keeps db.operation.name, and a cache call its command. A service keeps up to 32
operations of one target an hour; further ones count together as Other, under the same target.
Requests that ran out of time (a timeout error type, gRPC DEADLINE_EXCEEDED, HTTP 408 or 504) and
requests sent again (http.request.resend_count above 0) are counted beside the RED metrics.
Errors
MoleSignal classifies a span as an error when an exception is present, OpenTelemetry status isERROR, HTTP status is 500 or higher, or RPC status is non-zero. Error groups use
a stable fingerprint based on error type, the first application stack frame, and transaction name.
The changing error message is not part of the fingerprint. Grouping rules
can merge errors into one group or keep them out of groups.
Understand RED metrics
A service is a warning from a 2% error rate or a 500 ms p95, and critical from 10% or 2 s,
unless APM settings draw other lines. The service page and
the dot in the catalog say which figure passed which line.
The Insights card says which services moved against the stretch of the same length before and
against the same stretch a week before; see What insights find.
Latency histograms split each doubling of latency into eight buckets, from 125 µs to about 131 s, and a
percentile is read inside its bucket. A percentile is within about 4% of the exact value. Aggregates
written with the earlier fixed buckets stay readable at their coarser precision.
The Compare with control on the overview and the endpoint drawer draws the same range a day or a
week before as a dashed line, and the latency distribution outlines the spread of that range. On a
service page it also lists the service’s other environments: the same range in that environment is
drawn dashed instead.
The slowest requests of an endpoint come from the trace store, so every listed request opens its
trace. An error sample can point to a trace that tail sampling dropped; the trace drawer then says
that the trace was not found.
Service catalog
The catalog lists every service APM sees, one row per service and environment, and every service declared before it reports. Declare a service in the catalog and the Settings section of a service page record:
Archive in the same form takes the service out of the catalog’s default view. Its data is still
collected and its page still opens. Archived in the catalog lists archived services too.
Catalog requirements ask a team, an owner, or a repository
of the services from a tier on. A service short of one reads Needs with what it lacks in the
catalog.
A definition is keyed by namespace and name, so it holds in every environment. The Columns menu
of the catalog and of the endpoint table chooses and orders the columns, and the arrangement is kept
for each user. Besides the figures and the catalog fields, the catalog offers Type,
Language / framework, Instances, and Region, read from the instances in the range, and
Tags and Last seen.
The star of a catalog row keeps the service among the reader’s favorites, which come first in the
catalog; Favorites lists them alone. Favorites are kept for each user. Group groups the rows by
team, namespace, tier, or the value of a tag. The catalog filters by environment and team, and by
namespace and tag where services have more than one. The filters and the grouping are part of the
address, so a shared link opens the same view. A service seen in the range that had not reported for a
day by the end of the range reads Inactive.
Key transactions are the endpoints that matter most. The star of an endpoint in the endpoint list or in
its drawer marks it, and Settings lists the service’s key transactions with what each served in the
range. A service keeps up to 50.
Service documents and runbooks live in the Mole Agent knowledge base under
services/<service name>/,
in the organization’s base or the owning team’s base. Docs on a service page lists them and starts a
runbook. Mole Agent reads the same files when it investigates the service.
Writing definitions, key transactions, and error states requires apm.manage. Owners, administrators,
and editors hold it.
Instances
Each process that reports for a service is an instance, told apart byservice.instance.id, else
k8s.pod.name, else host.name. MoleSignal looks at an instance once a minute of its spans and keeps
its version and the attributes in the table above, each the value last reported. A repository link is
kept as an https address without any user, password, or query. What each instance served (requests,
errors, and latency) is counted in buckets of its own, from the same spans as the service’s figures.
Runtime lists the instances of the range and version in view, the most recently seen first, with
throughput, error rate, and p95, and links to the instance’s logs, by its pod, else its host, and to its
metrics, by the label it is told apart by. Past the instance cardinality limit per service and hour,
the rest are counted together as Other instances, so the instances add up to the service. Clicking
an instance keeps the runtime charts to it, and the choice is part of the address. An instance not
seen for as long as hourly aggregates are kept is removed.
Endpoints in depth
The endpoint drawer adds two cards to the RED charts:- Status codes counts the responses by code: HTTP codes from
http.response.status_code(http.status_codein older instrumentation) and gRPC codes fromrpc.grpc.status_code, each colored by class, with their trend. A bucket keeps up to 24 codes; further codes count as Other HTTP or Other gRPC. Aggregates written before status codes were counted have none. - Where the time goes reads the latest requests of the endpoint in the range from the trace store,
at most 200, and the spans under each. A call’s time is set down to what it calls: a database
(
db.system.name), a cache (Redis, Memcached, Valkey), HTTP, RPC, or messaging, and Other for a call that says nothing of its target. Internal spans count as internal work, and time no span under the request covers is its own time. Each part is shown with its share and its time per request. Calls made in parallel can add up past a request’s length; the parts are then scaled to it. Only the traces that tail sampling kept are read.
Deployments
The Deployments section of a service page puts the deployments that pipelines report beside the versions that the service’s spans carry, names rollbacks, tells each version against the version before it, and reads two versions running at once side by side. Pipelines report deployments withPOST /api/v1/apm/deployments. See Deployments.
Dependencies
Each target on a service’s Dependencies tab has a health score from 100: four points come off for each percent of failed calls (at most 60), four for each percent of calls that ran out of time (at most 20), and, while a stretch is compared, half a point for each percent the P95 grew past a fifth (at most 20). From 90 a target is good, from 70 worth a look. The score’s title says what took points off. With Compare with set, each target also says how its error rate and P95 moved. Slow SQL lists the database statements with the highest P95, as templates.Investigate a performance problem
1
Find the service
Open APM → Services and sort by error rate or p95. Narrow by environment or team.
2
Read the overview
Open the service. Compare requests, error rate, and latency with a day or a week before, and read
where the time goes downstream. The Insights card says what moved against the stretch before and
the same stretch a week before, since when, and the changes near then. See
Insights and investigations.
3
Narrow the cause
Use Endpoints for slow entry points, Dependencies for downstream time, and Errors for
recurring failures. Sort endpoints by share of time when latency and traffic are both important.
4
Open evidence
Open an endpoint’s slowest request, an error sample, or a trace from Traces. The trace opens in a
drawer over the page. Logs, Metrics, and Profiles at the end of the service’s scope row
open those signals for the same service and range.
5
Check the deployment
Open Deployments and look for a regressed version. Select its verdict, or choose two versions,
then review the deltas and the endpoints and errors that got worse. Treat a not-enough-requests
result as inconclusive.
6
Hand it to Mole Agent
Click Investigate in the service header, in an endpoint or error drawer, or on a finding. Mole
Agent opens with the question drafted and the service attached. See
Investigate with Mole Agent.
Track backend errors
Error groups are open until someone sets them Investigating, Resolved, or Ignored. Each can be given to a team and assigned to a member of the workspace. A resolved group that occurs again reads as Regressed. APM → Errors shows the unresolved groups by default: open, under investigation, and regressed. Above the list, Where they happened counts the occurrences of the listed groups by version, environment, and the language of the service; an error group’s drawer counts its own. Related logs in the drawer opens Logs on the service and range, and Logs beside a sample opens the logs of the sample’s trace.Group errors with rules
Grouping rules on the errors list sets how errors are grouped beyond their type, frame, and endpoint. A rule names a service, or every service, and patterns for the error type, the message, the frame the error was thrown at, and the endpoint. Each pattern matches the whole text, ignoring case, with* for any text; an empty pattern matches anything, and a rule needs at least one.
Rules are tried in order and the first that matches holds; move a rule up or down to change the
order. A rule applies to the errors seen after it is saved, on every node within half a minute, and
leaves the groups already made as they are. While a rule is edited, the form says how many listed
groups fit it. Rule from this error in an error group’s drawer starts a
rule with the group’s type and service.
Service level objectives
A service level objective (SLO) promises a share of good requests over a rolling window of 7, 28, or 30 days. It covers a service in one environment, or one endpoint of it. Open APM → SLOs and click New SLO, or click New SLO in the Service level objectives card of a service overview.
An SLO reads the APM aggregates of its service, so it counts every request before tail sampling.
A latency objective reads the latency histograms, so the count of requests within the threshold is
as precise as a percentile, within about 4%. The window reads hourly aggregates, and burn rates read
minute aggregates. A window longer than the retention of hourly aggregates counts only the retained
days.
Burn-rate alerts
Create burn-rate alerts in an SLO’s drawer adds the two alert rules recommended by the Google SRE workbook. Both are APM alert rules on the objective and notify through the rule’s escalation policy.
The short window stops a rule from firing on a spike that is already over. Once a rule watches the
SLO, the drawer lists it in place of the button. Creating the rules requires
alerts.manage.
Writing SLOs requires apm.manage. Deleting an SLO deletes its burn-rate alerts first, which also
requires alerts.manage.
Alert on APM figures
An APM alert rule watches a figure of a service or one of its endpoints, read from the same aggregates as the APM pages, so it needs no query.1
Open the rule editor
Open New alert in the header of a service page or of an endpoint drawer and choose Threshold
alert. The editor opens on the APM alert source with the service, environment, and endpoint filled in. A new rule in
Alerts → Alert rules offers the same source as APM alert. Creating a rule requires
alerts.manage.2
Choose the figure
Choose Service, Environment or Every environment, Endpoint or Whole service,
and Figure. Set thresholds in the unit the editor names under the figure.
3
Test and save
Click Test rule to draw the figure over the last six hours against the thresholds, then
create the rule.
Every evaluation reads the rule’s window: minute aggregates up to 24 hours, and hourly aggregates
for a longer window. A window without requests reads
0 for Requests, Errors, New error groups,
and Throughput, so a rule can catch traffic that stopped. The other figures have no value then, no
threshold matches, and an open incident resolves.
A rule made in the editor or from an SLO carries the labels service.name, service.namespace,
deployment.environment, and apm.endpoint for what it names, and its incidents copy them. The
incidents of every APM rule also carry the service label, the service in affected_services, and a
sample trace ID from the aggregates. The Alerts column of the catalog, the service header, and the endpoint drawer count the open and
acknowledged incidents that name the service, in its environment when an incident names one. The
endpoint drawer counts only the incidents labeled with its endpoint. A count opens Alerts →
Incidents.
Anomaly alert in the same menu alerts when a figure strays from the same time on the days before,
with no threshold to set. See
Alert on anomalies of APM figures.
Filters, resolution, and retention
APM pages share the global time range. The catalog filters by environment and team, and a service page by environment and version. Lists support search and sorting. The scope is encoded in the URL for sharing or revisiting the view. With default deployment settings:- an omitted API range uses the last 24 hours;
autoresolution uses minute buckets for ranges up to 24 hours and hourly buckets for longer ranges;- minute aggregates are retained for 24 hours;
- hourly aggregates and the maximum query range cover 30 days.
Read data-quality notices
Every APM response includes data-quality metadata. When there is something to say, a short notice stands in the navigation card and opens its details.
Reduce the time range and filters first. If partial or delayed states persist, inspect the APM
health endpoint and platform telemetry.
Runtime metrics
The Runtime section charts the runtime metrics a service sends with itsservice.name: the
process’s figures, which any language has, then the runtime of each language the service reports in
telemetry.sdk.language, or every runtime with metrics when it reports none. Each chart draws one
line per instance, told apart as above, and reads both OpenTelemetry’s conventions, as stored here, and
the Prometheus client each runtime commonly uses. A chart uses the first of its metrics the service
sends:
The heap and non-heap charts of the JVM read
jvm.memory.type (OpenTelemetry) or area (Micrometer).
A counter is drawn as a rate a second over five minutes. Go’s scheduling latency is the mean over
five minutes, the sum of its histogram over its count.
Profiling at the end of the section merges the service’s profiles of the range into a flame graph,
one profile type at a time, such as CPU, wall clock, allocations, heap, or locks. Open in Profiles
continues on the Profiles page.
Privacy and bounded data
APM stores aggregates and a small amount of sanitized evidence. APM does not retain request or response bodies, URL query values, headers, or SQL parameters in APM aggregates; a SQL statement is kept only as its template, every literal a?.
Identity values, representative messages, and stack frames are length-bounded. Sensitive-looking
and volatile values are removed or masked before persistence.
These protections do not replace good instrumentation hygiene. Avoid placing secrets or personal
data in span names, status descriptions, exception messages, or resource attributes.
Troubleshooting
APM API
Query APM catalogs, details, error states, service definitions, version comparisons, and health.
Alerting
Route APM incidents through escalation policies and notification channels.
Traces
Inspect the individual spans and traces behind APM evidence.
Service map
Visualize cross-service parent and child relationships.
Trace intake
Send OpenTelemetry traces to MoleSignal.