Before upgrading
- Read the release notes of every version between the current one and the target.
- Back up PostgreSQL and the object store. See Backup and restore.
- Note the running version.
molesignal --versionprints it, andGET /api/v1/versionreturns the version, commit, build id, release channel, and edition. The request needs a bearer token. - Rehearse the upgrade against a restored copy of the database when the deployment is critical.
How schema changes apply
The binary embeds its database migrations and applies them to PostgreSQL on every start, before it serves traffic. A migration failure stops the process. A lock in PostgreSQL serializes migrations, so several nodes that start at the same time are safe. The applied versions are recorded in the_molesignal_schema_meta table.
Migrations only move forward:
- A binary that is older than the database refuses to start with
migration <version> was previously applied but is missing in the resolved migrations. - A binary that carries a changed copy of an applied migration refuses to start with
migration <version> was previously applied but has been modified. Pre-release builds can change their baseline schema, so a database created by one build is not guaranteed to open with another.
Procedures
- Docker Compose
- Binary and systemd
- Kubernetes
Edit the Keep the named volumes. They hold PostgreSQL data, RustFS data, and the WAL.
image tag in the Compose file, then pull and recreate the container:SIGTERM or SIGINT waits up to [node].drain_timeout_secs for pending data to flush, and then exits. Records that remain in the WAL are replayed when the node starts.
Role-split clusters
A role-split cluster shares one schema. A running node of an older version keeps using the new schema until it restarts, and the versions are not guaranteed to be compatible. Upgrade all nodes in one maintenance window:- Stop the
routernodes, then theintakeandqueriernodes, thencompactorandalert. - Start one node of the new version and wait for
GET /api/v1/healthzto return200. That node applies the migrations. - Start the remaining nodes.
Verify the upgrade
- The log shows no migration error, and
GET /api/v1/healthzreturns200. GET /api/v1/versionreturns the new version.- A query over recent data returns rows, and new records arrive.
/metricsshows no growth ofstorage_reconciler_issues_total.