Skip to main content
Most incidents start with a change. A release that is marked in the data can be compared with the version before it within minutes, while a release nobody recorded is found only after an hour of guessing. Record every release, compare it while it rolls out, and agree beforehand on what makes it roll back.

​
Make every release visible

Use the same version string everywhere: the tag of the image, service.version, the deployment report, and the RUM version.

​
Know the baseline before releasing

Before the rollout, note how the service behaves now: the error rate and P95 of its main endpoints, the burn rate of its SLOs, and the open RUM issues. Run the synthetic checks of the critical paths with Run now, so a failure after the release is not mistaken for an old one. Write the rollback criteria down, for example: the version is Regressed on errors, a critical synthetic check fails twice, or the fast burn-rate alert fires.

​
Compare while it rolls out

  • While two versions run at once, the Canaries card of the Deployments section reads them side by side: the share of requests each one serves, errors, and latency, with the verdict of the new version.
  • Each version gets a verdict against the version before it: Regressed, Improved, No change, or Not enough requests. A comparison needs enough requests, 1,000 by default; keep a canary long enough to get them.
  • Select a verdict to open Compare two versions, which lists the endpoints that got slower and the errors that grew.
  • On RUM, filter the explorer and the error issues by the new version, and watch for issues marked Regressed: an issue resolved earlier that came back in a newer version.

​
Let automation watch the release

An automation of Mole Agent started by the CI job can take the first look: the deploy step calls its trigger with a description of the change, and Mole Agent follows the instructions of the automation, for example to compare the new version with the old one, and reports in a chat. Keep a person responsible for the decision.

​
Decide and record

Roll back when a criterion is met, without waiting for certainty. Report the rollback from CI with status set to rolled_back, so the timeline shows it and later comparisons use the right baseline. After a release that stays, check the SLOs over the next day: slow regressions, such as a memory leak or a growing queue, often appear only hours later.

​
Acceptance criteria

  • Every service sends service.version, and every production deployment appears in the deployment timeline with its commit.
  • Frontend releases upload source maps or symbols for the same version they report.
  • Rollback criteria are written down before the rollout.
  • Each release gets a verdict, or a canary long enough to reach enough requests.
Last modified on October 26, 2026