> ## Documentation Index
> Fetch the complete documentation index at: https://docs.molesignal.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Verify a release

> Make every release visible in APM and RUM, compare the new version with the old one while it rolls out, and decide quickly whether to keep it or roll back.

Most incidents start with a change. A release that is marked in the data can be compared with the
version before it within minutes, while a release nobody recorded is found only after an hour of
guessing. Record every release, compare it while it rolls out, and agree beforehand on what makes it
roll back.

## Make every release visible

| Step | How | What it enables |
| - | - | - |
| Version every span | Set `service.version` in the OpenTelemetry resource of each service. | Versions in the [deployment timeline](https://docs.molesignal.com/en-US/apm/deployments.md#the-deployment-timeline) and verdicts per version, even without CI reporting. |
| Report the deployment from CI | `POST /api/v1/apm/deployments` with the service, version, environment, commit, and CI run, using the default intake token. See [Report deployments from CI](https://docs.molesignal.com/en-US/apm/deployments.md#report-deployments-from-ci). | The commit, author, and CI run beside the version, and failed or rolled-back deployments. |
| Report configuration and flags | The same endpoint with `kind` set to `configuration` or `feature_flag`. | Changes that are not releases appear near APM findings. |
| Version the frontend | The `version` of the RUM SDK, and the source maps or symbols uploaded from the same CI job. See [Source maps and symbols](https://docs.molesignal.com/en-US/rum/source-maps.md). | Readable stack traces, and RUM issues that know in which version they came back. |

Use the same version string everywhere: the tag of the image, `service.version`, the deployment report,
and the RUM version.

## Know the baseline before releasing

Before the rollout, note how the service behaves now: the error rate and P95 of its main endpoints, the
burn rate of its [SLOs](https://docs.molesignal.com/en-US/apm#service-level-objectives), and the open RUM issues. Run the
[synthetic checks](https://docs.molesignal.com/en-US/synthetics.md) of the critical paths with **Run now**, so a failure after the
release is not mistaken for an old one.

Write the rollback criteria down, for example: the version is **Regressed** on errors, a critical
synthetic check fails twice, or the fast burn-rate alert fires.

## Compare while it rolls out

* While two versions run at once, the **Canaries** card of the
  [Deployments](https://docs.molesignal.com/en-US/apm/deployments.md#canaries) section reads them side by side: the share of requests
  each one serves, errors, and latency, with the verdict of the new version.
* Each version gets a verdict against the version before it: **Regressed**, **Improved**, **No change**,
  or **Not enough requests**. A comparison needs enough requests, 1,000 by default; keep a canary long
  enough to get them.
* Select a verdict to open **Compare two versions**, which lists the endpoints that got slower and the
  errors that grew.
* On RUM, filter the explorer and the error issues by the new version, and watch for issues marked
  **Regressed**: an issue resolved earlier that came back in a newer version.

## Let automation watch the release

An [automation](https://docs.molesignal.com/en-US/automations.md#start-it-from-ci/cd) of Mole Agent started by the CI job can take the
first look: the deploy step calls its trigger with a description of the change, and Mole Agent follows the
instructions of the automation, for example to compare the new version with the old one, and reports in a
chat. Keep a person responsible for the decision.

## Decide and record

Roll back when a criterion is met, without waiting for certainty. Report the rollback from CI with
`status` set to `rolled_back`, so the timeline shows it and later comparisons use the right baseline.

After a release that stays, check the SLOs over the next day: slow regressions, such as a memory leak or
a growing queue, often appear only hours later.

## Acceptance criteria

* Every service sends `service.version`, and every production deployment appears in the deployment
  timeline with its commit.
* Frontend releases upload source maps or symbols for the same version they report.
* Rollback criteria are written down before the rollout.
* Each release gets a verdict, or a canary long enough to reach enough requests.

## Related guides

* [Deployments](https://docs.molesignal.com/en-US/apm/deployments.md)
* [RUM errors and issues](https://docs.molesignal.com/en-US/rum/errors.md)
* [Automations](https://docs.molesignal.com/en-US/automations.md)
* [Watch a critical user journey](https://docs.molesignal.com/en-US/best-practices/user-journeys.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.