> ## Documentation Index > Fetch the complete documentation index at: https://docs.molesignal.com/llms.txt > Use this file to discover all available pages before exploring further. # Role-based deployment > Run MoleSignal roles in separate processes: what each role does, how nodes find each other, and the current limits of a role-split cluster. MoleSignal is a single binary. The `[node].roles` setting decides which responsibilities a process carries. The default, `standalone`, runs every role in one process. Splitting roles lets intake, query, and background work scale and fail independently. The `router` role is the entry point of a role-split cluster. It serves the complete HTTP API and the web UI, forwards writes to intake nodes, and sends queries to querier nodes. Every other role serves a small internal HTTP listener for health checks and metrics. ## Roles | Role | What it runs | State | Scaling | | - | - | - | - | | `standalone` | Every role in one process, the full HTTP API, the web UI, internal gRPC, OTLP gRPC, and the Probe Agent gRPC endpoints | Stateful: embeds an intake with a WAL | One process. Not a horizontal scaling unit. | | `router` | The complete HTTP API and web UI, the OTLP gRPC receiver, forwarding of writes to intake nodes, forwarding of queries to querier nodes, and per-organization rate limiting | Stateless | Scales horizontally behind a load balancer. | | `intake` | WAL replay, in-memory buffering, periodic flush to Parquet, real-time alert rules, the internal gRPC intake endpoint, and the OTLP gRPC receiver | Stateful: the WAL of the most recent flush window | Sharded by stream with rendezvous hashing. Plan the WAL volume and the rebalancing before scaling. | | `querier` | Query execution for SQL and PromQL requests that other nodes send, the Arrow Flight scan endpoint for federated queries, and asynchronous search jobs | Stateless | Scales horizontally. | | `compactor` | File compaction, retention, and storage maintenance: index rebuild, delayed object deletion, and catalog reconciliation | Stateless between runs | Scales horizontally. Several nodes divide the work by stream and by organization. See [Work division between compactor nodes](#work-division-between-compactor-nodes). | | `alert` | Rule evaluation, escalation dispatch, scheduled reports, scheduled log inspections, connector runners, pipeline runners, background jobs that run on one node, and the Probe Agent gRPC endpoints | Stateless between runs: evaluation state and incidents live in the metadata database | Scales horizontally. Several nodes divide the work by rule, report, pipeline, and connector, and one primary node runs the jobs that must run once. See [Work division between alert nodes](#work-division-between-alert-nodes). | A process can carry several roles, for example `["intake", "querier"]`. ### Listeners | Role | `[http].port` (5080) | Internal gRPC (5082) | OTLP gRPC (4317) | Other | | - | - | - | - | - | | `standalone` | Full HTTP API and web UI | Yes | Yes | Probe Agent ports `5084` and `5085`. Flight SQL on `5083` when enabled. | | `router` | Full HTTP API and web UI | No | Yes | Flight SQL on `5083` when enabled. | | `intake` | Health, readiness, and metrics | Yes | Yes | | | `querier` | Health, readiness, and metrics | Yes | No | Flight SQL on `5083` when enabled. | | `compactor` | Health, readiness, and metrics | No | No | | | `alert` | Health, readiness, and metrics | No | No | Probe Agent ports `5084` and `5085`. | The health, readiness, and metrics listener of a node without the full API answers three unauthenticated paths and nothing else: | Path | Answer | | - | - | | `GET /api/v1/healthz` | `200`, or `503` with a `reason` when a subsystem is degraded. | | `GET /api/v1/readyz` | `503` until WAL replay has finished, then `200`. | | `GET /metrics` | Prometheus metrics. | Keep this listener on the internal network. ## Select roles Set roles in the configuration file or with environment variables. Environment variables use the `MS_` prefix and join every level of the configuration path with a single underscore: `[node].roles` becomes `MS_NODE_ROLES`, and `[cluster].advertise_addr` becomes `MS_CLUSTER_ADVERTISE_ADDR`. Names with dots are not recognized. **TOML** ```toml theme={null} [node] id = "intake-a1" roles = ["intake"] [cluster] advertise_addr = "10.0.1.21:5082" heartbeat_interval_secs = 5 peer_timeout_secs = 15 ``` **Environment variables** ```bash theme={null} export MS_NODE_ROLES='["intake"]' export MS_NODE_ID=intake-a1 export MS_CLUSTER_ADVERTISE_ADDR=10.0.1.21:5082 export MS_SELF_TELEMETRY_CLUSTER_TOKEN="" molesignal --config ./conf/config.toml ``` **One process per role** ```bash theme={null} MS_NODE_ROLES='["router"]' MS_NODE_ROLES='["intake"]' MS_NODE_ROLES='["querier"]' MS_NODE_ROLES='["compactor"]' MS_NODE_ROLES='["alert"]' ``` Every node of a cluster must use the same PostgreSQL database and the same object storage, and must share these settings: `MS_CIPHER_KEY`, `MS_AUTH_JWT_SECRET_OVERRIDE` (when a fixed signing secret is needed), and `MS_SELF_TELEMETRY_CLUSTER_TOKEN`. Give every node a unique `[node].id` and an `advertise_addr` that its peers can reach on the internal gRPC port. Two processes that publish the same dataset under one `[node].id` fence each other. Once one of them has published, the flushes of the other are rejected. The fenced process drops the rows it still holds in memory, keeps its WAL on disk, and answers `503` for writes to that dataset until it restarts with a unique id. ## Data flow ### Intake path A request to an intake endpoint of the router (`/api/v1/intake/*`, or OTLP gRPC) is authenticated and decoded on the router. The router then forwards the batch over the internal gRPC port to an intake node chosen by rendezvous hashing of the organization and the stream over the ids of the live intake nodes. Every router sends the records of one stream to the same node, and when that node cannot be reached, every router tries the same next one, up to three nodes. A node that joins or leaves moves only its own streams. All the writes of one stream go to one node, so give a large source several streams to spread its writes. The router answers `503` when no intake node is alive. The intake node runs the pipelines, masking, and schema evolution, appends each batch to the WAL, then to an in-memory buffer, and answers the router with the result that a standalone node would give. A background flush encodes the buffer into Parquet files and a search index, uploads them to the object store, records them in the file catalog in PostgreSQL, and truncates the flushed WAL segments. Objects at least `[object_store].multipart_threshold_size` large are uploaded in parts. A collector can also send OTLP gRPC directly to an intake node. The other roles write the data they generate themselves, such as derived datasets, with the same WAL and flush on their own node. ![Intake path](https://mintcdn.com/molesignal/W03b-Z-TATDejvIA/images/architecture/intake_en_light.svg?fit=max\&auto=format\&n=W03b-Z-TATDejvIA\&q=85\&s=40be47750e410cebbf593a16145a5460) Intake path The `[wal]` and `[intake]` sections of [Configuration](https://docs.molesignal.com/en-US/configuration.md) control the WAL directory, the segment size, the flush strategy, the buffer size, and the flush interval. ### Query path A SQL or PromQL request reaches the router, which checks authorization, admission, and the result cache, then sends the request over the internal gRPC port to a querier node chosen in turn. The querier runs the complete query against its own engine and returns the result. The router tries the other queriers when the chosen one cannot be reached, and answers `503` with `no querier node is available` when none is alive. A query that names remote clusters is merged from their results on the router. ![Query path](https://mintcdn.com/molesignal/NJoXrfidjTl0w4wV/images/architecture/query_en_light.svg?fit=max\&auto=format\&n=NJoXrfidjTl0w4wV\&q=85\&s=58541f70efb3bd6380a72ff646af5f45) Query path A query reads the files that the file catalog lists. In a role-split cluster, a record that an intake node still buffers becomes visible to queries after the next flush, within `[intake].flush_interval_secs` (30 seconds by default) or sooner when the buffer reaches its size limit. ### Asynchronous search jobs A query whose estimated size exceeds `[querier].auto_async_threshold_rows` becomes a durable search job. The router stores the job in PostgreSQL, and workers on the querier nodes claim jobs without running the same job twice. See `[search_jobs]` in [Configuration](https://docs.molesignal.com/en-US/configuration.md). ![Async search job pipeline](https://mintcdn.com/molesignal/LKKF1DKLWF5Aik4g/images/architecture/async_en_light.svg?fit=max\&auto=format\&n=LKKF1DKLWF5Aik4g\&q=85\&s=9a4d7a9995a2a1f36d55c9939de5c494) Async search job pipeline ### Federated query Add `?clusters=local,` to a query to include remote clusters. The node that receives the request scans its own part, requests a scan from every enabled remote cluster with a bearer token, merges the batches, and runs the full SQL. A remote cluster that cannot be reached is reported as degraded in the federation metadata instead of silently becoming local data. Remote clusters are managed in **Settings**. ![Federated query](https://mintcdn.com/molesignal/VFApFmnpGfjQnMkz/images/architecture/federated_en_light.svg?fit=max\&auto=format\&n=VFApFmnpGfjQnMkz\&q=85\&s=149d89d451af70ba49eae3de826f759d) Federated query ## Cluster membership There is no gossip or consensus component. Every node writes a heartbeat row to PostgreSQL with its id, roles, and `advertise_addr`, every `[cluster].heartbeat_interval_secs` (default 5). Peers whose last heartbeat is older than `[cluster].peer_timeout_secs` (default 15) are ignored, and rows that stay silent for five minutes are removed. A node finds peers by reading that table and filtering by role, and caches the result for up to 2 seconds. A `standalone` node handles writes and queries in its own process and forwards nothing. A node that receives `SIGTERM` or `SIGINT` deletes its own row before it finishes its remaining work, so peers stop using it within 2 seconds instead of waiting for the timeout. A drain through `POST /api/v1/node/drain` deletes the row at the next heartbeat. The internal gRPC port `5082` carries write forwarding, query forwarding, and scan traffic between nodes without authentication. Keep it on a trusted network. ### Work division between compactor nodes Several nodes can run the `compactor` role at the same time. Each stream and each organization has exactly one owner among the live compactor nodes, and the owner runs the work of that stream or organization. A `standalone` node counts as one of the compactor nodes. Deletions that are safe to repeat run on every node. | Work | Divided by | Runs on | | - | - | - | | File compaction, metric downsampling, and retention | Stream | The owner of the stream | | Partition manifest sealing | Stream | The owner of the stream | | Index rebuild and catalog reconciliation | Organization | The owner of the organization | | Delayed object deletion | Delete queue entry | Every compactor node. A node claims an entry with a lease of `[storage.gc].lease_secs` before it deletes the object. | | Expiry of session replay segments and investigation stacks | Not divided | Every compactor node. Deleting an expired item again has no effect. | The owner comes from rendezvous hashing over the ids of the live compactor nodes. Nodes that see the same members compute the same owner, without a coordinator or a lock table. When a node joins or leaves, only the streams and organizations of that node move. * **Node failure.** A node that stops sending heartbeats leaves the membership after `[cluster].peer_timeout_secs`. Node discovery is cached for up to 2 seconds, so with the default settings the remaining nodes take over its streams and organizations about 17 seconds after the last heartbeat, and process them in the next round of each worker. * **Node shutdown.** A node that receives `SIGTERM` or `SIGINT` leaves the membership first and then finishes its in-flight work. The remaining nodes take over its streams and organizations about 2 seconds later, and process them in the next round of each worker. * **New node.** A node starts taking work after it has been in the membership for `[cluster].peer_timeout_secs`. The other nodes hand over its share as soon as they see it, so that share waits for one timeout window instead of being processed twice. * **Overlap.** Nodes read the membership at slightly different moments, so two nodes can briefly process the same stream while the membership changes. Every result is published to the file catalog in one atomic step, and the node that loses discards its output. An overlap costs some I/O and never duplicates or loses data. Set a unique `[node].id` on every node. Nodes that share an id count as one member, and each of them processes the whole share of that member. Every compactor node applies `[compactor].max_concurrent_groups` on its own, so the total merge concurrency grows with the number of nodes. On `/metrics`, the gauges `work_ownership_members`, `work_ownership_active`, and `work_ownership_owned_units` report the number of members a node sees, whether the node currently takes work, and how many streams or organizations it owned in its latest round. The `work` label of `work_ownership_owned_units` names the job: `compaction` and `partition_manifest` count streams, and `index_rebuild` and `storage_reconciler` count organizations. When the membership is stable, the values of all compactor nodes add up to the number of streams or organizations. ### Work division between alert nodes Several nodes can run the `alert` role at the same time. The nodes divide the work by item, and one node, the primary, runs the jobs that must run once. A `standalone` node counts as one of the alert nodes. | Work | Divided by | Runs on | | - | - | - | | Rule evaluation | Rule | The owner of the rule | | Escalation of open incidents | Rule of the incident | The owner of the rule, which is also the node that evaluates it | | Scheduled reports | Report | The owner of the report | | Scheduled pipelines | Pipeline | The owner of the pipeline | | Pull connectors | Connector | The owner of the connector | | Notification delivery, on-call events, and status page notifications | Not divided | Every alert node. A node claims an event or a delivery in the database before it sends it, and event ids are stable, so repeating the work has no effect. | | Scheduled log inspections | Not divided | Every alert node. A node claims a run in the database before it starts, so each run is made by one node. | | Cross-cluster event push and gossip, service graph recompute, APM rollup, status page maintenance, synthetic monitoring automation, slow query analysis, and incident root cause analysis | Not divided | The primary node only | The owner of an item comes from rendezvous hashing over the ids of the live alert nodes, in the same way as for compactor nodes. The primary is the owner of one fixed key. Membership changes follow the compactor rules above: the remaining nodes take over the items of a node that fails or shuts down with the same timings, and a new node takes work after it has been in the membership for `[cluster].peer_timeout_secs`. When the primary fails or shuts down, another node becomes the primary, and its jobs resume at their next round. A node that joins can take over the primary role, and the jobs pause until that node has finished its warm-up. * **Rules.** A node evaluates the rules it owns in each round (`[alert].eval_interval_secs`). During a membership change, two nodes can briefly own the same rule. A node skips a rule that was evaluated less than half of `[alert].eval_interval_secs` ago, so a period counts once and the `for_periods` debounce is not used up early. * **Reports, pipelines, and connectors.** A node claims an item with a lease before it runs the item, and releases the lease when the run ends. A node that crashes during a run keeps the lease for up to 10 minutes, then another node runs the item again. A run that takes longer than 10 minutes can be repeated by another node. A connector resumes from its stored cursor, so no data is lost, but the events of an interrupted window can be written twice. Keep the clocks of the nodes synchronized, because the nodes compare the timestamps that they write. Set a unique `[node].id` on every node. On `/metrics`, the `work` label of `work_ownership_owned_units` reports `rule_evaluation` and `escalation_dispatch` as the number of rules and incidents the node owned in its latest round, and `primary` as `1` on the primary node and `0` on the others. The `primary` value updates every `[cluster].heartbeat_interval_secs`. `work_overlap_skipped_total` counts the items that a node skipped because another node had already run them in the same period. The counter stays flat while the membership is stable. ## Rate limiting The `router` role limits requests per organization and route class with `[router.rate_limit]`. The intake class covers `/api/v1/intake/*` and the query class covers `/api/v1/query` and the paths below it. The limit counts requests of the authenticated organization and answers `429` with a `Retry-After` header when it is exceeded. Each router replica counts on its own, so the effective limit scales with the number of replicas. A `standalone` node does not apply this limit. ## Scaling and high availability * **Router.** Add replicas behind a load balancer. * **Querier.** Add replicas. Each query runs on one querier, so more queriers raise the number of queries that run at the same time. * **Intake.** Run each node with its own persistent WAL volume (a StatefulSet in Kubernetes). When the number of nodes changes, the streams of the node that joins or leaves move, so make sure the WAL has been flushed before scaling down, and leave time for WAL replay before routing traffic to a restarted node. * **Compactor.** Add replicas. The nodes divide the work by stream and by organization, the remaining nodes take over when one fails, and rolling updates are safe. See [Work division between compactor nodes](#work-division-between-compactor-nodes). * **Alert.** Add replicas. The nodes divide the work by rule, report, pipeline, and connector, one primary node runs the jobs that must run once, the remaining nodes take over when a node fails, and rolling updates are safe. See [Work division between alert nodes](#work-division-between-alert-nodes). * **Multi-node consensus.** WAL replication between intake nodes is not implemented. Losing an intake node's volume loses the records that were not yet flushed. Changing `[node].roles` requires a restart. See [Deploy MoleSignal](https://docs.molesignal.com/en-US/deployment) for installation and ports, and [Production checklist](https://docs.molesignal.com/en-US/deployment/production-checklist.md) before going live. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.