[node].roles setting decides which responsibilities a process carries. The default, standalone, runs every role in one process. Splitting roles lets intake, query, and background work scale and fail independently.
The router role is the entry point of a role-split cluster. It serves the complete HTTP API and the web UI, forwards writes to intake nodes, and sends queries to querier nodes. Every other role serves a small internal HTTP listener for health checks and metrics.
Roles
A process can carry several roles, for example
["intake", "querier"].
Listeners
The health, readiness, and metrics listener of a node without the full API answers three unauthenticated paths and nothing else:
Keep this listener on the internal network.
Select roles
Set roles in the configuration file or with environment variables. Environment variables use theMS_ prefix and join every level of the configuration path with a single underscore: [node].roles becomes MS_NODE_ROLES, and [cluster].advertise_addr becomes MS_CLUSTER_ADVERTISE_ADDR. Names with dots are not recognized.
- TOML
- Environment variables
- One process per role
MS_CIPHER_KEY, MS_AUTH_JWT_SECRET_OVERRIDE (when a fixed signing secret is needed), and MS_SELF_TELEMETRY_CLUSTER_TOKEN. Give every node a unique [node].id and an advertise_addr that its peers can reach on the internal gRPC port.
Two processes that publish the same dataset under one [node].id fence each other. Once one of them has published, the flushes of the other are rejected. The fenced process drops the rows it still holds in memory, keeps its WAL on disk, and answers 503 for writes to that dataset until it restarts with a unique id.
Data flow
Intake path
A request to an intake endpoint of the router (/api/v1/intake/*, or OTLP gRPC) is authenticated and decoded on the router. The router then forwards the batch over the internal gRPC port to an intake node chosen by rendezvous hashing of the organization and the stream over the ids of the live intake nodes. Every router sends the records of one stream to the same node, and when that node cannot be reached, every router tries the same next one, up to three nodes. A node that joins or leaves moves only its own streams. All the writes of one stream go to one node, so give a large source several streams to spread its writes. The router answers 503 when no intake node is alive.
The intake node runs the pipelines, masking, and schema evolution, appends each batch to the WAL, then to an in-memory buffer, and answers the router with the result that a standalone node would give. A background flush encodes the buffer into Parquet files and a search index, uploads them to the object store, records them in the file catalog in PostgreSQL, and truncates the flushed WAL segments. Objects at least [object_store].multipart_threshold_size large are uploaded in parts.
A collector can also send OTLP gRPC directly to an intake node. The other roles write the data they generate themselves, such as derived datasets, with the same WAL and flush on their own node.
Intake path
[wal] and [intake] sections of Configuration control the WAL directory, the segment size, the flush strategy, the buffer size, and the flush interval.
Query path
A SQL or PromQL request reaches the router, which checks authorization, admission, and the result cache, then sends the request over the internal gRPC port to a querier node chosen in turn. The querier runs the complete query against its own engine and returns the result. The router tries the other queriers when the chosen one cannot be reached, and answers503 with no querier node is available when none is alive. A query that names remote clusters is merged from their results on the router.
Query path
[intake].flush_interval_secs (30 seconds by default) or sooner when the buffer reaches its size limit.
Asynchronous search jobs
A query whose estimated size exceeds[querier].auto_async_threshold_rows becomes a durable search job. The router stores the job in PostgreSQL, and workers on the querier nodes claim jobs without running the same job twice. See [search_jobs] in Configuration.
Async search job pipeline
Federated query
Add?clusters=local,<cluster-name> to a query to include remote clusters. The node that receives the request scans its own part, requests a scan from every enabled remote cluster with a bearer token, merges the batches, and runs the full SQL. A remote cluster that cannot be reached is reported as degraded in the federation metadata instead of silently becoming local data. Remote clusters are managed in Settings.
Federated query
Cluster membership
There is no gossip or consensus component. Every node writes a heartbeat row to PostgreSQL with its id, roles, andadvertise_addr, every [cluster].heartbeat_interval_secs (default 5). Peers whose last heartbeat is older than [cluster].peer_timeout_secs (default 15) are ignored, and rows that stay silent for five minutes are removed. A node finds peers by reading that table and filtering by role, and caches the result for up to 2 seconds. A standalone node handles writes and queries in its own process and forwards nothing.
A node that receives SIGTERM or SIGINT deletes its own row before it finishes its remaining work, so peers stop using it within 2 seconds instead of waiting for the timeout. A drain through POST /api/v1/node/drain deletes the row at the next heartbeat.
The internal gRPC port 5082 carries write forwarding, query forwarding, and scan traffic between nodes without authentication. Keep it on a trusted network.
Work division between compactor nodes
Several nodes can run thecompactor role at the same time. Each stream and each organization has exactly one owner among the live compactor nodes, and the owner runs the work of that stream or organization. A standalone node counts as one of the compactor nodes. Deletions that are safe to repeat run on every node.
The owner comes from rendezvous hashing over the ids of the live compactor nodes. Nodes that see the same members compute the same owner, without a coordinator or a lock table. When a node joins or leaves, only the streams and organizations of that node move.
- Node failure. A node that stops sending heartbeats leaves the membership after
[cluster].peer_timeout_secs. Node discovery is cached for up to 2 seconds, so with the default settings the remaining nodes take over its streams and organizations about 17 seconds after the last heartbeat, and process them in the next round of each worker. - Node shutdown. A node that receives
SIGTERMorSIGINTleaves the membership first and then finishes its in-flight work. The remaining nodes take over its streams and organizations about 2 seconds later, and process them in the next round of each worker. - New node. A node starts taking work after it has been in the membership for
[cluster].peer_timeout_secs. The other nodes hand over its share as soon as they see it, so that share waits for one timeout window instead of being processed twice. - Overlap. Nodes read the membership at slightly different moments, so two nodes can briefly process the same stream while the membership changes. Every result is published to the file catalog in one atomic step, and the node that loses discards its output. An overlap costs some I/O and never duplicates or loses data.
[node].id on every node. Nodes that share an id count as one member, and each of them processes the whole share of that member.
Every compactor node applies [compactor].max_concurrent_groups on its own, so the total merge concurrency grows with the number of nodes.
On /metrics, the gauges work_ownership_members, work_ownership_active, and work_ownership_owned_units report the number of members a node sees, whether the node currently takes work, and how many streams or organizations it owned in its latest round. The work label of work_ownership_owned_units names the job: compaction and partition_manifest count streams, and index_rebuild and storage_reconciler count organizations. When the membership is stable, the values of all compactor nodes add up to the number of streams or organizations.
Work division between alert nodes
Several nodes can run thealert role at the same time. The nodes divide the work by item, and one node, the primary, runs the jobs that must run once. A standalone node counts as one of the alert nodes.
The owner of an item comes from rendezvous hashing over the ids of the live alert nodes, in the same way as for compactor nodes. The primary is the owner of one fixed key. Membership changes follow the compactor rules above: the remaining nodes take over the items of a node that fails or shuts down with the same timings, and a new node takes work after it has been in the membership for
[cluster].peer_timeout_secs. When the primary fails or shuts down, another node becomes the primary, and its jobs resume at their next round. A node that joins can take over the primary role, and the jobs pause until that node has finished its warm-up.
- Rules. A node evaluates the rules it owns in each round (
[alert].eval_interval_secs). During a membership change, two nodes can briefly own the same rule. A node skips a rule that was evaluated less than half of[alert].eval_interval_secsago, so a period counts once and thefor_periodsdebounce is not used up early. - Reports, pipelines, and connectors. A node claims an item with a lease before it runs the item, and releases the lease when the run ends. A node that crashes during a run keeps the lease for up to 10 minutes, then another node runs the item again. A run that takes longer than 10 minutes can be repeated by another node. A connector resumes from its stored cursor, so no data is lost, but the events of an interrupted window can be written twice.
[node].id on every node.
On /metrics, the work label of work_ownership_owned_units reports rule_evaluation and escalation_dispatch as the number of rules and incidents the node owned in its latest round, and primary as 1 on the primary node and 0 on the others. The primary value updates every [cluster].heartbeat_interval_secs. work_overlap_skipped_total counts the items that a node skipped because another node had already run them in the same period. The counter stays flat while the membership is stable.
Rate limiting
Therouter role limits requests per organization and route class with [router.rate_limit]. The intake class covers /api/v1/intake/* and the query class covers /api/v1/query and the paths below it. The limit counts requests of the authenticated organization and answers 429 with a Retry-After header when it is exceeded. Each router replica counts on its own, so the effective limit scales with the number of replicas. A standalone node does not apply this limit.
Scaling and high availability
- Router. Add replicas behind a load balancer.
- Querier. Add replicas. Each query runs on one querier, so more queriers raise the number of queries that run at the same time.
- Intake. Run each node with its own persistent WAL volume (a StatefulSet in Kubernetes). When the number of nodes changes, the streams of the node that joins or leaves move, so make sure the WAL has been flushed before scaling down, and leave time for WAL replay before routing traffic to a restarted node.
- Compactor. Add replicas. The nodes divide the work by stream and by organization, the remaining nodes take over when one fails, and rolling updates are safe. See Work division between compactor nodes.
- Alert. Add replicas. The nodes divide the work by rule, report, pipeline, and connector, one primary node runs the jobs that must run once, the remaining nodes take over when a node fails, and rolling updates are safe. See Work division between alert nodes.
- Multi-node consensus. WAL replication between intake nodes is not implemented. Losing an intake node’s volume loses the records that were not yet flushed. Changing
[node].rolesrequires a restart.