Browse traces and spans
The Traces tab lists one row per request: how it ended (the HTTP status of its root span, or OK and Error), when it started, its service and operation, its duration against the longest trace of the page, its span count, its error count, and its trace ID. Click Started, Duration, Spans, or Errors to sort by that column. Over the table, a histogram counts every trace the query matches, and a chart beside it draws the p50, p95, and p99 duration of those traces. The Spans tab lists one row per matching span, with its kind and its own duration. A filter on a field of a span holds for the span itself there. The query editor has the modes Code and Builder. Code takes a field query, such asstatus_code = 'ERROR' or duration_ns > 500000000 (longer than 500 ms), or SQL against the trace
stream. Builder shows the field query as rows of filters, one row for each alternative. Queries run in tabs that can stand side by
side, the field list opens a field analysis, and Share copies a link to the query, as on
Logs. The address bar holds the applied query as well, so a copy of it
shares the query (Share a query).
A comparison on a field of a span keeps the traces with a span that matches it, and != keeps the
traces none of whose spans has the value. trace_id, duration_ns, span_count, and error_count
are compared on the trace itself. in asks for any of several values:
service.name in ('checkout', 'payments').
AND and OR join comparisons, AND first, and brackets group them, as on
Logs. A comparison keeps its meaning inside an OR:
service.name = 'checkout' OR span_count > 50 keeps the traces with a span of checkout and the
traces of more than 50 spans. Text without a field, such as timeout, is looked for in the trace
ID, span ID, service, operation, and status. The Spans tab and the RED drilldown read the same
query for each span: a span matches a OR b when it matches a or b.
The field list stands beside the results. Common fields come first: service.name, name,
status_code, the HTTP status, and duration_ns. A field opens on the values the loaded traces
hold. Filter for this value adds the value to the query, a query of alternatives put in brackets
first so the value narrows all of it, and a second value of the same field
joins the first in an in list, so several values can be ticked. Filter out this value leaves a
value out.
Columns over the Traces table shows, hides, and reorders its columns, and adds fields of
each trace’s root span as columns; the eye of such a field in the field list does the same. The
arrangement is kept per stream for each user, and Reset to default brings back the columns the
table opens with.
Inspect a trace
Click a row to open the trace in a drawer over the list. Drag the left edge of the drawer to change its width, which is kept for the next trace. J and K move to the next and previous trace of the list, and the drawer opens the trace’s own page as well. The head of the trace stays in place: the root service and operation, the HTTP status, the error count, the duration, the start time, the numbers of spans and services, the depth, and the trace ID. Related logs and Service metrics open those signals around the trace. Profiles opens the profiles that kept the trace’s ID, or the profiles of the minutes around the trace when none did. User session opens the RUM session the trace served: the session a span names insession.id or session_id, or else the session whose browser request carried the
trace’s ID in traceparent.
Download trace at the end of the head saves the trace as a file:
Four views show the spans:
Critical path in the toolbar of the waterfall and the flame graph draws the spans that the
trace’s end-to-end time waited on. The path starts at the end of the root span and goes back in
time: at each moment it follows the child span that ended last by then, cut to the span that called
it, and goes on from the moment that child began. A child still running at that moment ran alongside
the one the path followed and is not on the path. Time that no child covers stays with the calling
span. On the waterfall, a line along each bar
marks the stretches of that span on the path, and the other spans fade. The legend gives the length
of the path and the share of it that each service holds. The switch stays on from trace to trace.
The detail of the selected span stands beside the view, and its width can be dragged. It has the
tabs Attributes, Resource, Events, Links, Logs, Metrics, and Profile.
Links also lists spans of other traces that link to the span. Logs shows the log lines that
carry the trace ID, for the span or the whole trace. Metrics draws the requests, errors, and p95
duration of the span’s service and operation in the minutes around it.
Profile shows what the profiler sampled while the span ran. A sample is linked to a span when
the profiler records it with the trace context, as OTLP profiles do. With linked
samples, the tab lists the functions that took the most of one profile type by self value,
Open flame graph draws those samples, and Open in Profiles opens them on the Profiles page.
A flame mark on a waterfall row shows that the span has linked samples. Without linked samples,
Profiles around this span opens the profiles of the minutes around the span.
Each trace also has its own page, which a link to a trace opens. If a browser RUM session contains a
session identifier, open Session traces to review related backend requests.
Compare two traces
Click Pick two traces to compare over the Traces table, then click two rows. The first trace picked is A and the second is B. Click Compare to open the comparison on the page of A. In the head of a trace, Compare with another trace takes the trace ID of the other one. The comparison shows both traces, their durations, and how much longer or shorter B took than A. Three tabs follow, each with the biggest change first:
Self time is the time that spans spent on their own work, not the time spent waiting on the spans
that they called. Swap A and B and Stop comparing stand at the head of the page, and the
address of the page keeps both traces, so a comparison can be shared.
Service graph and catalog
The Service Graph tab derives nodes and edges from trace spans. The graph displays request rate, error rate, and latency relationships without a separately maintained topology. The Service Catalog groups observed services and operations for navigation from a service to current traces.Intake and propagation
Send traces with OTLP HTTP or OTLP gRPC. MoleSignal preserves standard W3C Trace Context and OpenTelemetry semantic attributes. Response headers exposeX-Request-Id and X-Trace-Id for
supported application requests.
Permissions
An organization workspace requiresstreams.query. Platform administrators use
sys.telemetry.read for the protected _sys telemetry scope.
Application performance monitoring
Analyze RED metrics, transactions, dependencies, errors, and deployments derived from spans.
Service map
Understand topology derived from cross-service parent and child spans.
Backend distributed tracing
Operate MoleSignal’s own tracing and tail-sampling pipeline.