Overview: design philosophy
Single binary
Every role is compiled into the same binary and packaged into the same image. Processes are distinguished purely by configuration (
[node].roles); no separate build artifacts are required.Roles selected by config
[node].roles determines process exposure and foreground responsibilities. The default is ["standalone"], meaning one process runs every responsibility.State externalized where possible
Metadata lands in Postgres, data lands in the object store. Aside from the Intake’s WAL/buffer, most roles are stateless and scale horizontally.
Discovery via Postgres
There is no separate gossip or consensus component. Each node writes a heartbeat into the
cluster_nodes table, and peer discovery queries that table directly.- Single-node standalone
- Role-split cluster
- Evaluation, development, PoC, low-traffic production.
- One process exposes HTTP + gRPC and runs every internal responsibility in-process (intake, query, compaction, alert evaluation, etc.).
- Dependencies are still externalized: Postgres + object store (or a local filesystem backend).
Role enum values use snake_case in TOML / environment variables (
standalone, alert_manager), matching the naming convention of the internal implementation. All config examples below use snake_case.Node role reference
[node].roles is an array (Vec<Role>), with valid values: standalone, router, intake, querier, compactor, alert_manager. The default is ["standalone"].
Multiple roles compose in one process. A form like
[node].roles = ["intake", "querier"] starts the deduplicated set of foreground servers required by each role, including the gRPC server serving intake and scan. The configuration gates matching background loops and registers the node under all configured roles in cluster_nodes, allowing peer discovery by role. standalone is shorthand for “every role in one process”.A standalone
querier or any role set containing querier or intake starts the gRPC server, which carries the Arrow Flight scan service. The coordinator discovers queriers via list_role(querier) and fans shards out over that RPC. A single-process standalone still works as before; distributed scan-out requires at least two querier peers.Which role carries which background worker
Role-specific loops (intake flush, compaction + file_meta_dumper, alert evaluation/dispatch) are spawned only when the owning role is configured (standalone counts as all roles). A handful of always-on workers (heartbeat, sweeper, object-store probe, MMDB refresh, search-jobs, scheduled-reports) run on every node. The table below gives the owning role and interval.
ACME issuance and renewal are implemented: the issue loop scans
pending domains, and the renewal loop renews active certificates within the 30-day window. Both loops use the single-issuance path with a per-domain cooldown. The TLS-terminating HTTP server starts the runner when [http.tls].enabled = true. TLS and ACME are compiled into every build without a feature flag and remain runtime-gated by [http.tls].enabled.How to select roles
Configure roles through[node].roles; environment variables can override the setting. Environment variables share the MS_ prefix, with section and field separated by a . (dot). MoleSignal strips the MS_ prefix and splits the remaining key on .. Examples: MS_NODE.ROLES, MS_STORE.META.DSN, MS_HTTP.PORT.
- TOML
- Env var override
- A set of role-split processes
A few bootstrap / secret variables are flat single-underscore and live outside
Settings, read directly from the environment: MS_CIPHER_KEY, MS_AUTH_JWT_SECRET_OVERRIDE, MS_LICENSE_FILE, MS_SELF_TELEMETRY_CLUSTER_TOKEN, and the development-only MS_AGENT_<PROVIDER>_* fallback. All other structured fields use the MS_<SECTION>.<FIELD> dotted form (for example MS_NODE.ROLES, MS_STORE.META.DSN, and MS_CLUSTER.ADVERTISE_ADDR).Data flow
Intake path
The entry point lands on an Intake through the Router with rate limiting and consistent hashing. The Intake first writes the WAL durably to disk, then writes the in-memory buffer. A background flush loop encodes the buffer into columnar files and a search index by time window or size threshold, uploads each file to the object store, stores FileMeta in Postgres, and truncates flushed WAL segments.
Intake path
[wal].dir, [wal].segment_size_mb, [wal].flush_strategy (batch/none/every_write), [wal].sync_level (data/all); [intake].buffer_max_mb (default 256), flush_interval_secs (default 30), flush_parallelism (default 4); [router.rate_limit].intake_qps (default 1000 per org, 0 = unlimited).
The Router rate limits at
(org_id, route_class) granularity, returning 429 with a Retry-After header when exceeded. org_id comes from the X-Org-Id request header and defaults to default.Query path
The query entry point is on a node that exposes HTTP. The engine is wrapped layer by layer according to cluster size: local query engine → (with ≥2 querier peers) distributed engine → (when a remote cluster is specified) federated engine.
Query path
Distributed query shards are fanned out to peers through a hash of
object_key; shard SQL only performs a scan (SELECT * FROM <stream>), while full aggregation runs on the coordinator to avoid partial/final aggregation inconsistencies. The distributed path runs only when the cluster has at least two querier peers; otherwise the query falls back to the local engine with no network hop.Async search job pipeline
Async search job pipeline
[search_jobs].workers (default 2), idle_poll_secs (default 2), cleanup_interval_secs (default 3600); the automatic async threshold [querier].auto_async_threshold_rows (default 50 million rows).
The
FOR UPDATE SKIP LOCKED claim semantics are safe across multiple workers / multiple nodes: multiple processes carrying search_jobs can share the same search_jobs table and claim concurrently without executing the same job twice.Federated / multi-cluster query
Target clusters are specified via?clusters=local,sf,nyc. After scanning locally, the coordinator issues one internal scan RPC (with a Bearer token) to each enabled remote cluster, then UNION ALLs the batches each cluster returns with the local data and executes the full SQL.
Federated / multi-cluster query
Cluster membership and discovery
1
Node registration
The heartbeat task periodically upserts
(node_id, roles, advertise_addr, last_heartbeat_at_micros) into the Postgres cluster_nodes table (primary key node_id, ON CONFLICT DO UPDATE). The full node role set is stored comma-joined, so a multi-role node occupies one row discoverable under every configured role. The first heartbeat fires immediately, followed by heartbeats at the configured interval.2
Heartbeat interval
[cluster].heartbeat_interval_secs, default 5s. advertise_addr defaults to 127.0.0.1:5082, i.e. the gRPC address (host:port) peers use to interconnect.3
Liveness window and stale cleanup
The liveness window is controlled by
[cluster].peer_timeout_secs, default 15s: the registry only returns nodes with last_heartbeat_at >= now - peer_timeout. Additionally, a sweeper deletes cluster_nodes rows that have not heartbeated for over 5 minutes every 60s.4
Peer discovery
There is no gossip or consensus. In distributed mode, each role queries the
cluster_nodes table directly and filters live peers by role membership. A node matches when the role set contains the requested role. standalone mode skips the entire discovery flow and returns only the local node.5
Placement algorithm
Router selecting an Intake: applies consistent hashing over
org_id|stream_name then mods, landing deterministically on a particular Intake. Router selecting a Querier: naive round-robin (now_ns % peer_count), not full consistent hashing. Distributed query sharding: a hash over object_key then mods, fanning out to querier peers.6
Distributed scan RPC
The coordinator encodes a scan request (org/stream/sql/file_metas/time_range) into a ticket and sends the ticket to peers over the internal scan RPC. Each peer reads columnar files, registers an in-memory table, runs the shard SQL, and returns a result stream. This trusted RPC shares port
5082 with node services and the private cluster intake protocol. Standard external OTLP gRPC uses 4317.Local scan-RPC calls within the cluster are unauthenticated; only federated / remote-cluster calls use an optional Bearer token.
cluster_nodes table schema: node_id VARCHAR(64) PK, role VARCHAR(128) (the comma-joined role set), advertise_addr VARCHAR(255), started_at_micros BIGINT, last_heartbeat_at_micros BIGINT. list_role scans alive rows and matches membership in code, so role lookups don’t depend on the column being indexed.
External dependencies and ports
PostgreSQL
The metadata database: FileMeta, streams, rules, incidents, users, orgs, audit, quotas, certificates,
cluster_nodes, search_jobs, remote_clusters, and more. [store.meta]: backend (default sqlite; set to postgres for production), dsn, max_connections (default 16). Migrations are embedded at compile time.Object store
Columnar files + search-index sidecars.
[store.object].backend: local (default, root=./data/objects) / s3 (including MinIO, R2, Alibaba Cloud OSS, overridden via endpoint) / azure / gcs. Credential precedence: env vars > credential file > inline TOML.No external cache / consensus
Only in-process LRU+TTL caches, the rate limiter, and the async runtime. No Redis / Memcached, no external consensus component.
Rendering dependency (optional)
PNG/PDF rendering for scheduled reports requires headless Chromium, an enabled renderer, and a reachable Web
base_url. If rendering is unavailable, the request returns an explicit error rather than a placeholder file.Listening ports
/metrics (GET, Prometheus text 0.0.4) is always mounted. Restrict the endpoint at the reverse proxy or network boundary. The endpoint exposes fixed-cardinality cache, object-store, WAL, query, alerting, and self-observability metrics.Health / readiness probe semantics
1
Object store probe at startup (blocking)
startup_ping() synchronously performs a PUT→GET→DELETE (128 bytes) against _health/{uuid}.probe. On failure, the process does not start.2
Background periodic probe
The same round-trip runs every
[store.object].health_probe_interval_secs (default 30s); only 3 consecutive failures set object_store_degraded=true, and a success resets the counter to zero.3
Intake WAL replay
At startup the Intake scans segment files under
[wal].dir, replays all records into the in-memory buffer by (org, stream_type, stream), forces a single flush after loading, then sets replay_done=true. /api/v1/readyz returns 503 until replay completes.Authentication and config overrides
- The JWT secret is auto-bootstrapped by the database on first startup; the old
jwt_secretTOML field is deprecated (parsing is retained only for backward compatibility with old configs). To pin a fixed secret, use the environment variableMS_AUTH_JWT_SECRET_OVERRIDE. - API tokens take the form
ms_<prefix>_<secret>, with the secret stored as an argon2id hash. - At startup, fields such as
store.meta.dsn,wal.dir,http.port,grpc.port, andnode.idare treated as immutable; runtime changes trigger a warning.
Deployment topologies
All topologies share the same image (e.g.molesignal:dev), distinguished by MS_NODE.ROLES. The web frontend is a separate nginx image (e.g. molesignal-web:dev).
- Single-node sandbox
- Docker Compose
- Kubernetes
A single process with all roles combined; the fastest way to get started.
The local backend (
store.object.backend=local) is fine for development; use an object store backend for production.Scaling and high availability
Router — scales horizontally
Stateless; scale replicas with entry traffic. The rate limiter is in-process and ephemeral: under multiple replicas each replica counts independently, so the effective org QPS ceiling is roughly
configured value × replica count. When needed, move rate limiting up to a unified gateway or lower the per-replica threshold.Querier — scales horizontally
Stateless. A querier process serves the scan RPC over gRPC; distributed scans only trigger with ≥2 querier peers, otherwise the coordinator falls back to the in-process engine with no network hop. The bottleneck is columnar-file reads + query-engine memory.
Intake — stateful
Use a StatefulSet + a per-replica PVC for a persistent WAL. The Router places via consistent hashing over
org|stream; scaling changes the mod result and causes rebalancing, so ensure the WAL is flushed before scaling down. Leave enough of a replay window for the readiness probe (the manifest sets a 10s initial delay).Compactor / AlertManager — single instance
Both are recommended at a single replica: multiple Compactor instances would produce merge conflicts (a lease-table lock is planned for the future), and AlertManager’s evaluation / incident state is in-process. Reliability comes from fast restarts rather than multiple replicas.
wal_append_lock_wait_seconds, wal_append_inflight, wal_fsync_errors_total, file_meta_dump_*; for the object store, object_store_operations_total, object_store_errors_total, object_store_op_duration_seconds, object_store_probe_*; for queries, the cache hit rate cache_* and tantivy_pruned_files_total; for alerts, alert_rule_eval_timeout_total. Combine /api/v1/healthz, /api/v1/readyz, and the cluster_nodes table to observe member liveness.
Minimal production checklist / validation checklist
1
External dependencies ready
Postgres reachable,
[store.meta].backend = "postgres" with a correct dsn; choose an object store backend of s3/azure/gcs (not local), with credentials injected in the precedence order “env vars > credential file > inline”.2
Roles and interconnect addresses
Each process has an explicit
MS_NODE.ROLES; set MS_CLUSTER.ADVERTISE_ADDR to a peer-reachable host:5082 (K8s uses $(POD_IP):5082). A process may carry several roles at once, such as ["intake","querier"]. The process starts the deduplicated server set and registers under every configured role.3
Intake persistence
Run the Intake as a StatefulSet + PVC mounting the WAL directory; set
[wal].flush_strategy / sync_level according to durability requirements; leave enough replay delay for the readiness probe.4
Single-instance roles
Keep Compactor and AlertManager at 1 replica each. Confirm
[compactor].interval_secs, retention_days, and [storage.file_meta_dump].enabled match expectations.5
Entry point and rate limiting
Put an LB in front of the Router; set
[router.rate_limit].intake_qps / query_qps per org; note that rate limiting is approximate under multiple replicas. Disable buffering for /api/v1/query/stream on the Ingress.6
Observability and probes
Wire
/metrics into Prometheus; point K8s probes at /api/v1/healthz (liveness) and /api/v1/readyz (readiness). Confirm the object store startup probe passes (otherwise the process will not start).7
Security and license
Set
MS_AUTH_JWT_SECRET_OVERRIDE when a fixed JWT secret is required; otherwise the database bootstraps the secret automatically. Inject a cipher key and avoid the development all-zero value in production. Federated query requires the federated_search license feature. TLS and automatic certificate issuance or renewal are available in every build without a feature flag and activate at runtime through [http.tls].enabled.