Skip to main content
MoleSignal keeps state in PostgreSQL, in object storage, on the intake WAL volume, and in configuration and secrets. Back up each of them, and rehearse a restore before relying on the backups.

​
What to back up

​
Keep the database and the objects consistent

The file catalog in PostgreSQL references objects in the bucket. A flush uploads an object first and records it in the catalog afterwards. Compaction and retention remove a catalog reference first, and the garbage collector deletes the object later, after [storage.gc].grace_period_secs (default 3600 seconds). Use this order for a consistent backup:
  1. Take the PostgreSQL backup.
  2. Copy the bucket right afterwards, and finish within the grace period.
A copy that completes within the grace period contains every object that the database backup references. If the copy takes longer, raise grace_period_secs before the backup, or enable bucket versioning and restore the objects as of the time of the database backup.

​
Back up PostgreSQL

Use the backup features of the PostgreSQL service, such as scheduled snapshots with point-in-time recovery. For a self-managed database, a logical dump works for small deployments:

​
Back up object storage

Use bucket replication or versioning of the provider. A scheduled copy to a second bucket also works:
For an S3-compatible service such as RustFS, add --endpoint-url. Back up with credentials that cannot delete from the backup bucket.

​
Restore

1

Stop every node

Stop all MoleSignal processes so that nothing writes while the data is replaced.
2

Restore PostgreSQL

Restore the backup into an empty database:
3

Restore the objects

Copy the backup into the bucket that the nodes use. Restore the bucket to the time of the database backup or later.The reconciler queues objects that the restored catalog does not reference as orphans, and the garbage collector deletes them after [storage.reconciler].orphan_grace_secs (default 24 hours). Copy newer objects elsewhere first if they may still be needed.
4

Restore secrets and configuration

Provide the configuration and the same MS_CIPHER_KEY that the backed-up deployment used. A node with a different key cannot decrypt the stored secrets and fails to start with an error such as open Probe CA key.
5

Restore the WAL volume if available

Mount the WAL volume at [wal].dir. Intake replays it on start. Without the volume, records that were not flushed at the time of the failure are lost.
6

Start and verify

Start the nodes and wait for GET /api/v1/healthz to return 200. Sign in, open a stream, and run a query over the restored range.Check /metrics for storage_reconciler_issues_total with reason="missing". Such entries mean that the catalog references objects that are not in the bucket.

​
Failure scenarios

Last modified on October 9, 2026