> ## Documentation Index
> Fetch the complete documentation index at: https://docs.molesignal.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Metrics, traces, and services

> Run PromQL with sparklines, browse metric names and labels, find traces and draw one as a waterfall, sum up APM services, compare two versions, and run long queries as search jobs with the MoleSignal CLI.

## Query metrics

`molesignal metrics query` runs a PromQL range query and prints one line per series: its last,
lowest, and highest value over the window, a sparkline of it, and its labels.

```bash theme={null}
molesignal metrics query 'sum by (service) (rate(http_server_requests_total[5m]))' --since 6h
```

```text theme={null}
LAST   MIN    MAX    TREND                           SERIES
12.4   3.1    14.9   ▁▂▂▃▅▆▇█▇▆▆▅▅▆▇▇█▇▆▅▄▄▅▅▆▆▇▇▇▆   {service="checkout"}
```

The window is the last hour by default; `--since`, `--from`, and `--to` change it. `-n` caps the
points, `1000` by default. `--json` prints each series with its points, and `--csv` prints the
points as rows.

Four more subcommands find what to query:

| Command | Lists |
| - | - |
| `molesignal metrics names [text]` | Metric names, those that contain the text. |
| `molesignal metrics labels <metric>` | The labels of a metric. |
| `molesignal metrics values <metric> <label>` | The values of a label, with how often each occurs. |
| `molesignal metrics series <metric>` | The series of a metric; `--match label=value` narrows them. |

## Find traces

```bash theme={null}
molesignal traces --service checkout --errors --since 6h
```

The list shows the newest traces with their start, duration, number of spans, and service.

| Option | Meaning |
| - | - |
| `--service <name>` | Only traces of this service. |
| `--errors` | Only traces with a failed span. |
| `--min-duration <time>` | Only traces that took at least this long, such as `500ms`. |
| `--since`, `--from`, `--to` | The window, the last hour by default. |
| `-n`, `--limit <count>` | How many traces, `20` by default and `200` at most. |

## Draw a trace

`molesignal traces show` draws one trace as a waterfall: one line per span, under its parent,
with a bar for when it ran within the trace and how long it took. Failed spans are marked with
`!`, and red on a terminal.

```bash theme={null}
molesignal traces show 4bf92f3577b34da6a3ce929d0e0e4736
```

```text theme={null}
checkout POST /orders          ████████████████████████████████████████  1.24s
├─ checkout SELECT orders       ███                                       98ms
└─ ! payments POST /charge         ███████████████████████████████████    1.1s
```

`--web` opens the trace in MoleSignal, and `--json` prints its spans.

## Sum up services

```bash theme={null}
molesignal services --env production --sort errors
```

The list shows each APM service with its requests, error rate, p95 latency, health, and current
version. `--sort` orders it by `requests`, `errors`, `p95`, or `name`.

`molesignal services show` sums up one service: its rate, errors, and latency, then its
endpoints, dependencies, errors, and versions:

```bash theme={null}
molesignal services show checkout --since 2h
```

## Compare two versions

`molesignal services compare` sets two versions of a service side by side and says whether the
candidate regressed, improved, held, or had too few requests to tell. With
`--fail-on-regression`, a regression ends the command with `1`, so that a pipeline can stop a
release on it:

```bash theme={null}
molesignal services compare checkout 2.3.9 2.4.0 --since 1h --fail-on-regression
```

## Run long queries as search jobs

A search job runs a query in the background over more data and more time than a query on the
spot, and keeps its result for a while. Submitting and cancelling one go through the
organization's Tool Policy, like every change; see
[Incidents and changes](https://docs.molesignal.com/en-US/cli/changes.md#how-a-change-runs).

```bash theme={null}
molesignal search-jobs submit "SELECT * FROM app_logs WHERE level = 'error'" --since 30d --wait
```

The stream is the first one after `FROM`; `--stream` names another, and `--promql` submits a
PromQL statement, read at the end of the window, or at every `--step` such as `1m`. `--limit` caps
the rows the job keeps, at most `100000`, and `--keep` how long it keeps them. `--wait` waits for
the job to finish. The command prints the job's id.

| Command | Effect |
| - | - |
| `molesignal search-jobs` | Lists the search jobs with their state, rows, and statement. |
| `molesignal search-jobs show <job>` | Prints a job's state, rows, times, and error. |
| `molesignal search-jobs results <job>` | Writes a finished job's rows, one JSON object per line; `--csv` writes CSV and `-o <file>` writes to a file. |
| `molesignal search-jobs cancel <job>` | Cancels a pending or running job. |

## Access

Metrics and traces take `streams.query` or `sys.telemetry.read`, as do APM and search jobs. The
organization's Tool Policy must leave the tools on.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.