> ## Documentation Index
> Fetch the complete documentation index at: https://docs.molesignal.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Distributed & Role-Based Deployment

> Node-role matrix, background-worker ownership, data flow, cluster discovery, and three topologies: single-node / Docker Compose / Kubernetes.

MoleSignal is a **single binary**: the same executable and image serves every role. Process configuration selects the active roles. The binary supports both a one-command sandbox and horizontal scaling into a multi-role cluster.

This page targets operators and SREs. The guide covers role breakdown, background-worker ownership, data flow, cluster discovery, external dependencies and ports, and three deployment topologies: single-node, Docker Compose, and Kubernetes.

## Overview: design philosophy

<CardGroup cols={2}>
  <Card title="Single binary" icon="box">
    Every role is compiled into the same binary and packaged into the same image. Processes are distinguished purely by configuration (`[node].roles`); no separate build artifacts are required.
  </Card>

  <Card title="Roles selected by config" icon="sliders">
    `[node].roles` determines process exposure and foreground responsibilities. The default is `["standalone"]`, meaning one process runs every responsibility.
  </Card>

  <Card title="State externalized where possible" icon="database">
    Metadata lands in Postgres, data lands in the object store. Aside from the Intake's WAL/buffer, most roles are stateless and scale horizontally.
  </Card>

  <Card title="Discovery via Postgres" icon="network-wired">
    There is no separate gossip or consensus component. Each node writes a heartbeat into the `cluster_nodes` table, and peer discovery queries that table directly.
  </Card>
</CardGroup>

**When to run single-node vs. split by role:**

<Tabs>
  <Tab title="Single-node standalone">
    * Evaluation, development, PoC, low-traffic production.
    * One process exposes HTTP + gRPC and runs every internal responsibility in-process (intake, query, compaction, alert evaluation, etc.).
    * Dependencies are still externalized: Postgres + object store (or a local filesystem backend).
  </Tab>

  <Tab title="Role-split cluster">
    * Intake and query loads need to scale independently; or the stateful Intake needs to be scheduled separately from the stateless roles.
    * Use the Router as the entry point and rate limiter, fanning intake across multiple Intake nodes via consistent hashing and distributing queries across multiple Queriers.
    * A good fit for Kubernetes: stateless roles as Deployments, the Intake as a StatefulSet + PVC.
  </Tab>
</Tabs>

<Note>
  Role enum values use **snake\_case** in TOML / environment variables (`standalone`, `alert_manager`), matching the naming convention of the internal implementation. All config examples below use snake\_case.
</Note>

## Node role reference

`[node].roles` is an array (`Vec<Role>`), with valid values: `standalone`, `router`, `intake`, `querier`, `compactor`, `alert_manager`. The default is `["standalone"]`.

<Note>
  **Multiple roles compose in one process.** A form like `[node].roles = ["intake", "querier"]` starts the deduplicated set of foreground servers required by each role, including the gRPC server serving intake and scan. The configuration gates matching background loops and registers the node under **all** configured roles in `cluster_nodes`, allowing peer discovery by role. `standalone` is shorthand for "every role in one process".
</Note>

| Role            | Foreground exposure                                                                                                      | Stateful / stateless                                          | Scaling characteristics                                                                            |
| --------------- | ------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- |
| `standalone`    | HTTP + gRPC, running every internal responsibility in-process                                                            | Stateful (embeds an Intake)                                   | Single instance for evaluation; not suitable as a horizontal scaling unit                          |
| `router`        | HTTP reverse proxy + rate limiting; **does not directly expose internal services beyond the business HTTP listener**     | Stateless (the rate limiter is in-process and ephemeral)      | Scales horizontally; just put an L4/L7 LB in front                                                 |
| `intake`        | Carries WAL replay + buffering + periodic flush (the flush loop is spawned at startup when the role is configured)       | **Stateful** (WAL, in-memory buffer, file\_meta cache)        | Sharded via consistent hashing; scaling must account for the WAL persistent volume and rebalancing |
| `querier`       | gRPC distributed-scan endpoint (Arrow Flight `do_get`): reads columnar files, runs the shard SQL, streams result batches | Stateless                                                     | Scales horizontally                                                                                |
| `compactor`     | Runs the compaction + retention tick loop (spawned only when the role is configured)                                     | Stateless (processes tick by tick)                            | **Single instance recommended** (see the high-availability section)                                |
| `alert_manager` | Runs the evaluation + dispatch tick loops (spawned only when the role is configured)                                     | Stateful (evaluation state and incident state are in-process) | Single instance recommended                                                                        |

<Note>
  A standalone `querier` or any role set containing `querier` or `intake` starts the gRPC server, which carries the Arrow Flight scan service. The coordinator discovers queriers via `list_role(querier)` and fans shards out over that RPC. A single-process `standalone` still works as before; distributed scan-out requires at least two querier peers.
</Note>

### Which role carries which background worker

Role-specific loops (intake flush, compaction + file\_meta\_dumper, alert evaluation/dispatch) are **spawned only when the owning role is configured** (`standalone` counts as all roles). A handful of always-on workers (heartbeat, sweeper, object-store probe, MMDB refresh, search-jobs, scheduled-reports) run on every node. The table below gives the owning role and interval.

| Background worker                                    | Recommended carrying role                                                     | Interval                                                                                     | Config key                                                               | Status                                             |
| ---------------------------------------------------- | ----------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------ | -------------------------------------------------- |
| heartbeat                                            | All roles                                                                     | `heartbeat_interval_secs`, default 5s (first beat immediate)                                 | `[cluster].heartbeat_interval_secs`                                      | Wired up                                           |
| stale node cleanup (sweeper)                         | All roles                                                                     | Fixed 60s                                                                                    | None (hardcoded)                                                         | Wired up                                           |
| object store health probe                            | All roles (blocking probe once at startup)                                    | `health_probe_interval_secs`, default 30s                                                    | `[store.object].health_probe_interval_secs`                              | Wired up                                           |
| alert evaluation (evaluator)                         | `alert_manager`                                                               | `eval_interval_secs`, default 30s                                                            | `[alert_manager].eval_interval_secs`                                     | Wired up                                           |
| alert dispatch (dispatcher)                          | `alert_manager`                                                               | `dispatch_interval_secs`, default 10s                                                        | `[alert_manager].dispatch_interval_secs`                                 | Wired up                                           |
| compaction                                           | `compactor`                                                                   | `interval_secs`, default 300s (5 min)                                                        | `[compactor].interval_secs`                                              | Wired up                                           |
| file\_meta\_dumper (cold partition flush to storage) | `compactor` (same process as compaction)                                      | `interval_secs`, default 3600s (1h)                                                          | `[storage.file_meta_dump].interval_secs`; set `enabled=false` to disable | Wired up                                           |
| scheduled\_reports                                   | `alert_manager` (needs rendering dependencies, see below)                     | Fixed 60s tick; each report cron determines due state                                        | the report `cron` (the tick interval is not configurable)                | Wired up                                           |
| search\_jobs (async search pool)                     | The role carrying `AppState` (typically `standalone` / a node exposing query) | Idle poll `idle_poll_secs`, default 2s; cleanup `cleanup_interval_secs`, default 3600s       | `[search_jobs].workers` (default 2), etc.                                | Wired up                                           |
| ACME issuance / renewal                              | the TLS-terminating HTTP server (`router` / `standalone`)                     | Issuance `issue_poll_secs` (default 60s); renewal `renewal_retry_secs` (default 21600s / 6h) | `[http.tls].issue_poll_secs` / `renewal_retry_secs`                      | Wired up (active when `[http.tls].enabled = true`) |

<Note>
  ACME issuance and renewal are implemented: the issue loop scans `pending` domains, and the renewal loop renews `active` certificates within the 30-day window. Both loops use the single-issuance path with a per-domain cooldown. The TLS-terminating HTTP server starts the runner when `[http.tls].enabled = true`. TLS and ACME are compiled into every build without a feature flag and remain runtime-gated by `[http.tls].enabled`.
</Note>

## How to select roles

Configure roles through `[node].roles`; environment variables can override the setting. Environment variables share the `MS_` prefix, with **section and field separated by a `.` (dot)**. MoleSignal strips the `MS_` prefix and splits the remaining key on `.`. Examples: `MS_NODE.ROLES`, `MS_STORE.META.DSN`, `MS_HTTP.PORT`.

<Tabs>
  <Tab title="TOML">
    ```toml theme={null}
    [node]
    # Unique node identifier; leave empty to have the process generate one
    id = "intake-a1"
    # Role array; multiple roles compose in one process
    roles = ["intake"]

    [cluster]
    # gRPC address (host:port) peers use to interconnect
    advertise_addr = "10.0.1.21:5082"
    heartbeat_interval_secs = 5
    peer_timeout_secs = 15
    ```
  </Tab>

  <Tab title="Env var override">
    ```bash theme={null}
    # Variable names use the dotted form, matching the docker-compose / k8s manifests.
    # Note: a dot is not a valid POSIX shell identifier, so `export MS_NODE.ROLES=…` is invalid;
    # inject via the container / orchestrator env, or launch with an env prefix:
    env 'MS_NODE.ROLES=["intake"]' \
        'MS_NODE.ID=intake-a1' \
        'MS_CLUSTER.ADVERTISE_ADDR=10.0.1.21:5082' \
        'MS_HTTP.PORT=5080' \
        'MS_GRPC.PORT=5082' \
        molesignal --config ./conf/config.toml
    ```
  </Tab>

  <Tab title="A set of role-split processes">
    ```bash theme={null}
    # One env line per process (the value is the role that process carries):
    # Entry router
    MS_NODE.ROLES='["router"]'

    # Intake node (needs a persistent volume for the WAL)
    MS_NODE.ROLES='["intake"]'

    # Query node
    MS_NODE.ROLES='["querier"]'

    # Compaction + cold partition flush to storage
    MS_NODE.ROLES='["compactor"]'

    # Alert evaluation + dispatch + scheduled reports
    MS_NODE.ROLES='["alert_manager"]'
    ```
  </Tab>
</Tabs>

<Info>
  A few bootstrap / secret variables are **flat single-underscore** and live outside `Settings`, read directly from the environment: `MS_CIPHER_KEY`, `MS_AUTH_JWT_SECRET_OVERRIDE`, `MS_LICENSE_FILE`, `MS_SELF_TELEMETRY_CLUSTER_TOKEN`, and the development-only `MS_AGENT_<PROVIDER>_*` fallback. All other structured fields use the `MS_<SECTION>.<FIELD>` dotted form (for example `MS_NODE.ROLES`, `MS_STORE.META.DSN`, and `MS_CLUSTER.ADVERTISE_ADDR`).
</Info>

## Data flow

### Intake path

The entry point lands on an Intake through the Router with rate limiting and consistent hashing. The Intake first writes the WAL durably to disk, then writes the in-memory buffer. A background flush loop encodes the buffer into columnar files and a search index by time window or size threshold, uploads each file to the object store, stores FileMeta in Postgres, and truncates flushed WAL segments.

<Frame caption="Intake path">
  <img src="https://mintcdn.com/molesignal/W03b-Z-TATDejvIA/images/architecture/intake_en_light.svg?fit=max&auto=format&n=W03b-Z-TATDejvIA&q=85&s=40be47750e410cebbf593a16145a5460" alt="Intake path" className="block dark:hidden" width="898" height="792" data-path="images/architecture/intake_en_light.svg" />

  <img src="https://mintcdn.com/molesignal/W03b-Z-TATDejvIA/images/architecture/intake_en_dark.svg?fit=max&auto=format&n=W03b-Z-TATDejvIA&q=85&s=3bfd2c8e823fea3e2dcd8e6773a3d5eb" alt="Intake path" className="hidden dark:block" width="898" height="792" data-path="images/architecture/intake_en_dark.svg" />
</Frame>

Key config: `[wal].dir`, `[wal].segment_size_mb`, `[wal].flush_strategy` (`batch`/`none`/`every_write`), `[wal].sync_level` (`data`/`all`); `[intake].buffer_max_mb` (default 256), `flush_interval_secs` (default 30), `flush_parallelism` (default 4); `[router.rate_limit].intake_qps` (default 1000 per org, 0 = unlimited).

<Note>
  The Router rate limits at `(org_id, route_class)` granularity, returning `429` with a `Retry-After` header when exceeded. `org_id` comes from the `X-Org-Id` request header and defaults to `default`.
</Note>

### Query path

The query entry point is on a node that exposes HTTP. The engine is wrapped layer by layer according to cluster size: local query engine → (with ≥2 querier peers) distributed engine → (when a remote cluster is specified) federated engine.

<Frame caption="Query path">
  <img src="https://mintcdn.com/molesignal/LKKF1DKLWF5Aik4g/images/architecture/query_en_light.svg?fit=max&auto=format&n=LKKF1DKLWF5Aik4g&q=85&s=6a447df8476aebb21b92608f38ac3847" alt="Query path" className="block dark:hidden" width="569" height="1038" data-path="images/architecture/query_en_light.svg" />

  <img src="https://mintcdn.com/molesignal/LKKF1DKLWF5Aik4g/images/architecture/query_en_dark.svg?fit=max&auto=format&n=LKKF1DKLWF5Aik4g&q=85&s=35e5c13238a10b73a6c0c33c8306ef03" alt="Query path" className="hidden dark:block" width="569" height="1038" data-path="images/architecture/query_en_dark.svg" />
</Frame>

<Note>
  Distributed query shards are fanned out to peers through a hash of `object_key`; shard SQL only performs a scan (`SELECT * FROM <stream>`), while full aggregation runs on the coordinator to avoid partial/final aggregation inconsistencies. The distributed path runs only when the cluster has at least two querier peers; otherwise the query falls back to the local engine with no network hop.
</Note>

### Async search job pipeline

<Frame caption="Async search job pipeline">
  <img src="https://mintcdn.com/molesignal/LKKF1DKLWF5Aik4g/images/architecture/async_en_light.svg?fit=max&auto=format&n=LKKF1DKLWF5Aik4g&q=85&s=9a4d7a9995a2a1f36d55c9939de5c494" alt="Async search job pipeline" className="block dark:hidden" width="816" height="793" data-path="images/architecture/async_en_light.svg" />

  <img src="https://mintcdn.com/molesignal/LKKF1DKLWF5Aik4g/images/architecture/async_en_dark.svg?fit=max&auto=format&n=LKKF1DKLWF5Aik4g&q=85&s=31a0a30ca4dcf3f1695cb76f1406b7bc" alt="Async search job pipeline" className="hidden dark:block" width="816" height="793" data-path="images/architecture/async_en_dark.svg" />
</Frame>

Config: `[search_jobs].workers` (default 2), `idle_poll_secs` (default 2), `cleanup_interval_secs` (default 3600); the automatic async threshold `[querier].auto_async_threshold_rows` (default 50 million rows).

<Info>
  The `FOR UPDATE SKIP LOCKED` claim semantics are safe across multiple workers / multiple nodes: multiple processes carrying `search_jobs` can share the same `search_jobs` table and claim concurrently without executing the same job twice.
</Info>

### Federated / multi-cluster query

Target clusters are specified via `?clusters=local,sf,nyc`. After scanning locally, the coordinator issues one internal scan RPC (with a Bearer token) to each enabled remote cluster, then UNION ALLs the batches each cluster returns with the local data and executes the full SQL.

<Frame caption="Federated / multi-cluster query">
  <img src="https://mintcdn.com/molesignal/LKKF1DKLWF5Aik4g/images/architecture/federated_en_light.svg?fit=max&auto=format&n=LKKF1DKLWF5Aik4g&q=85&s=9d1b25930c495ebb34812282879a64ab" alt="Federated / multi-cluster query" className="block dark:hidden" width="540" height="680" data-path="images/architecture/federated_en_light.svg" />

  <img src="https://mintcdn.com/molesignal/LKKF1DKLWF5Aik4g/images/architecture/federated_en_dark.svg?fit=max&auto=format&n=LKKF1DKLWF5Aik4g&q=85&s=afff26dc50b0206beaf28ad995c4bcbe" alt="Federated / multi-cluster query" className="hidden dark:block" width="540" height="680" data-path="images/architecture/federated_en_dark.svg" />
</Frame>

<Warning>
  **Federated query is license-gated:** as long as `clusters` contains a non-`local` target and the current license lacks the `federated_search` feature, the HTTP layer returns `403` directly. OpenSource Edition remains single-cluster. Remote cluster definitions are stored in the Postgres `remote_clusters` table (`advertise_addr`, `token_secret_ref`, `tls_verify`, `enabled`); clusters with `enabled=false` are skipped during fan-out. Remote auth currently supports Bearer token only; `tls_verify=false` maps to `http://` (rather than "https with verification skipped").
</Warning>

## Cluster membership and discovery

<Steps>
  <Step title="Node registration">
    The heartbeat task periodically upserts `(node_id, roles, advertise_addr, last_heartbeat_at_micros)` into the Postgres `cluster_nodes` table (primary key `node_id`, `ON CONFLICT DO UPDATE`). The full node role set is stored comma-joined, so a multi-role node occupies one row discoverable under every configured role. The first heartbeat fires immediately, followed by heartbeats at the configured interval.
  </Step>

  <Step title="Heartbeat interval">
    `[cluster].heartbeat_interval_secs`, default 5s. `advertise_addr` defaults to `127.0.0.1:5082`, i.e. the gRPC address (host:port) peers use to interconnect.
  </Step>

  <Step title="Liveness window and stale cleanup">
    The liveness window is controlled by `[cluster].peer_timeout_secs`, default 15s: the registry only returns nodes with `last_heartbeat_at >= now - peer_timeout`. Additionally, a sweeper deletes `cluster_nodes` rows that have not heartbeated for over 5 minutes every 60s.
  </Step>

  <Step title="Peer discovery">
    There is no gossip or consensus. In distributed mode, each role queries the `cluster_nodes` table directly and filters live peers by **role membership**. A node matches when the role set contains the requested role. `standalone` mode skips the entire discovery flow and returns only the local node.
  </Step>

  <Step title="Placement algorithm">
    Router selecting an Intake: applies consistent hashing over `org_id|stream_name` then mods, landing deterministically on a particular Intake. Router selecting a Querier: naive round-robin (`now_ns % peer_count`), not full consistent hashing. Distributed query sharding: a hash over `object_key` then mods, fanning out to querier peers.
  </Step>

  <Step title="Distributed scan RPC">
    The coordinator encodes a scan request (org/stream/sql/file\_metas/time\_range) into a ticket and sends the ticket to peers over the internal scan RPC. Each peer reads columnar files, registers an in-memory table, runs the shard SQL, and returns a result stream. This trusted RPC shares port `5082` with node services and the private cluster intake protocol. Standard external OTLP gRPC uses `4317`.
  </Step>
</Steps>

<Info>
  Local scan-RPC calls within the cluster are unauthenticated; only federated / remote-cluster calls use an optional Bearer token.
</Info>

`cluster_nodes` table schema: `node_id VARCHAR(64) PK`, `role VARCHAR(128)` (the comma-joined role set), `advertise_addr VARCHAR(255)`, `started_at_micros BIGINT`, `last_heartbeat_at_micros BIGINT`. `list_role` scans alive rows and matches membership in code, so role lookups don't depend on the column being indexed.

## External dependencies and ports

<CardGroup cols={2}>
  <Card title="PostgreSQL" icon="elephant">
    The metadata database: FileMeta, streams, rules, incidents, users, orgs, audit, quotas, certificates, `cluster_nodes`, `search_jobs`, `remote_clusters`, and more. `[store.meta]`: `backend` (default `sqlite`; set to `postgres` for production), `dsn`, `max_connections` (default 16). Migrations are embedded at compile time.
  </Card>

  <Card title="Object store" icon="cloud">
    Columnar files + search-index sidecars. `[store.object].backend`: `local` (default, `root=./data/objects`) / `s3` (including MinIO, R2, Alibaba Cloud OSS, overridden via `endpoint`) / `azure` / `gcs`. Credential precedence: env vars > credential file > inline TOML.
  </Card>

  <Card title="No external cache / consensus" icon="memory">
    Only in-process LRU+TTL caches, the rate limiter, and the async runtime. No Redis / Memcached, no external consensus component.
  </Card>

  <Card title="Rendering dependency (optional)" icon="image">
    PNG/PDF rendering for scheduled reports requires headless Chromium, an enabled renderer, and a reachable Web `base_url`. If rendering is unavailable, the request returns an explicit error rather than a placeholder file.
  </Card>
</CardGroup>

### Listening ports

| Port | Protocol / service         | Config key                                                         | Notes                                                                                                       |
| ---- | -------------------------- | ------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------- |
| 5080 | HTTP                       | `[http].bind` (default `0.0.0.0`), `[http].port`                   | `/api/v1/*`, `/metrics`, `/api/v1/healthz`, `/api/v1/readyz`, `/.well-known/acme-challenge/`, etc.          |
| 5082 | Internal gRPC              | `[grpc].bind`, `[grpc].port`, `max_message_size_mb` (default 32MB) | Trusted node, scan, and private cluster intake protocols                                                    |
| 4317 | OTLP gRPC                  | `[otlp_grpc]`                                                      | External standard logs, metrics, traces, and profiles receiver; enabled by default on Standalone and Intake |
| 5083 | Flight SQL                 | `[flight_sql]`                                                     | External bearer-authenticated SQL listener; disabled by default on Standalone and Querier                   |
| 5084 | pprof HTTP                 | `[profiling]`                                                      | Node diagnostics; disabled and loopback-only by default                                                     |
| 80   | Plaintext HTTP in TLS mode | `[http.tls].plain_port`                                            | Health checks + ACME HTTP-01 challenge + 301 redirect to HTTPS                                              |
| 443  | HTTPS in TLS mode (SNI)    | `[http.tls].port`                                                  | Full routing; enabled at runtime by `[http.tls].enabled` (compiled into every build)                        |

<Note>
  **`/metrics`** (GET, Prometheus text 0.0.4) is always mounted. Restrict the endpoint at the reverse proxy or network boundary. The endpoint exposes fixed-cardinality cache, object-store, WAL, query, alerting, and self-observability metrics.
</Note>

### Health / readiness probe semantics

| Probe     | Path                  | 200 condition                                                                                                            | 503 condition                                          | Purpose                                                               |
| --------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------ | --------------------------------------------------------------------- |
| Liveness  | `GET /api/v1/healthz` | WAL replay complete (Intake-relevant only; other roles bypass the replay check) **and** the object store is not degraded | replay in progress **or** object store connection lost | Process liveness and dependency health                                |
| Readiness | `GET /api/v1/readyz`  | `replay_done = true` (even if the object store is degraded)                                                              | replay still in progress                               | Lets K8s keep admitting read traffic while the write path is degraded |

<Steps>
  <Step title="Object store probe at startup (blocking)">
    `startup_ping()` synchronously performs a PUT→GET→DELETE (128 bytes) against `_health/{uuid}.probe`. On failure, the process does not start.
  </Step>

  <Step title="Background periodic probe">
    The same round-trip runs every `[store.object].health_probe_interval_secs` (default 30s); only **3 consecutive failures** set `object_store_degraded=true`, and a success resets the counter to zero.
  </Step>

  <Step title="Intake WAL replay">
    At startup the Intake scans segment files under `[wal].dir`, replays all records into the in-memory buffer by `(org, stream_type, stream)`, forces a single flush after loading, then sets `replay_done=true`. `/api/v1/readyz` returns 503 until replay completes.
  </Step>
</Steps>

### Authentication and config overrides

* The JWT secret is auto-bootstrapped by the database on first startup; the old `jwt_secret` TOML field is deprecated (parsing is retained only for backward compatibility with old configs). To pin a fixed secret, use the environment variable `MS_AUTH_JWT_SECRET_OVERRIDE`.
* API tokens take the form `ms_<prefix>_<secret>`, with the secret stored as an argon2id hash.
* At startup, fields such as `store.meta.dsn`, `wal.dir`, `http.port`, `grpc.port`, and `node.id` are treated as immutable; runtime changes trigger a warning.

## Deployment topologies

All topologies share the same image (e.g. `molesignal:dev`), distinguished by `MS_NODE.ROLES`. The web frontend is a separate nginx image (e.g. `molesignal-web:dev`).

<Tabs>
  <Tab title="Single-node sandbox">
    A single process with all roles combined; the fastest way to get started.

    ```toml theme={null}
    [node]
    roles = ["standalone"]

    [store.meta]
    backend = "postgres"
    dsn = "postgres://molesignal:molesignal@localhost:5432/molesignal"

    [store.object]
    backend = "local"
    root = "./data/objects"

    [wal]
    dir = "./data/wal"
    ```

    ```bash theme={null}
    molesignal --config ./conf/config.toml
    # HTTP :5080  gRPC :5082
    ```

    <Note>The local backend (`store.object.backend=local`) is fine for development; use an object store backend for production.</Note>
  </Tab>

  <Tab title="Docker Compose">
    The Compose file lives at `deploy/docker/docker-compose.yaml` and depends on `postgres:5432` (with a health check) and `minio:9000`, with config mounted (read-only) from `../../conf/config.toml`. The image is built by `deploy/docker/Dockerfile` in three stages (frontend pnpm → Rust release → bookworm-slim runtime layer with chromium).

    **Standalone profile:**

    ```bash theme={null}
    docker compose -f deploy/docker/docker-compose.yaml --profile standalone up
    # A single molesignal container, MS_NODE.ROLES=["standalone"]
    # Exposes 5080(HTTP) / 5082(gRPC), mounts obs-wal:/data/wal
    ```

    The checked-in service does not publish external OTLP gRPC `4317`; add `4317:4317` when the
    host must reach that receiver.

    **multirole profile (after removing the legacy connector service):**

    ```bash theme={null}
    docker compose -f deploy/docker/docker-compose.yaml --profile multirole up
    # router / intake / querier / compactor / alert-manager
    ```

    * `molesignal-router`: exposes 5080, the entry point.
    * `molesignal-intake`: mounts `obs-wal:/data/wal` for a persistent WAL; the other roles are stateless.
    * Each remaining role container carries the corresponding responsibility.

    <Info>There is also `deploy/docker/Dockerfile.web` and `nginx.conf`: the web container reverse-proxies to `MS_BACKEND`, disables `proxy_buffering` for `/api/v1/query/stream` and relaxes read/write timeouts to 600s to support live tail; `/assets/*` (Vite hash-named) is set to 1-year immutable, and `index.html` is set to `no-store`.</Info>
  </Tab>

  <Tab title="Kubernetes">
    The manifests live in `deploy/k8s/` and implement the multirole topology. All core roles inject: `MS_NODE.ROLES`, `POD_IP` (fieldRef `status.podIP`), `MS_CLUSTER.ADVERTISE_ADDR=$(POD_IP):5082`, plus the object store secret, cipher key, and optional license file injected from a Secret.

    Deployment order: `00-namespace.yaml` → `10-configmap.yaml` → `20-secret.yaml` → the individual role manifests.

    | Manifest                | kind            | Replicas | Role            | Volumes / Service                                                                                                                                                            |
    | ----------------------- | --------------- | -------- | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
    | `30-router.yaml`        | Deployment      | 2        | `router`        | Stateless; 5080/http + 5082/grpc; ClusterIP; `readinessProbe GET /api/v1/healthz`                                                                                            |
    | `40-intake.yaml`        | **StatefulSet** | 2        | `intake`        | `volumeClaimTemplate` data 20Gi RWO/replica mounted at `/data` (WAL); headless Service (`clusterIP: None`); readiness probe with 10s initial delay (leaving a replay window) |
    | `50-querier.yaml`       | Deployment      | 2        | `querier`       | Stateless; 5080 + 5082 (scan RPC); ClusterIP; readiness probe 5s                                                                                                             |
    | `60-compactor.yaml`     | Deployment      | **1**    | `compactor`     | Stateless; single instance to avoid merge conflicts (a lease-table lock is planned for the future); ClusterIP                                                                |
    | `70-alert-manager.yaml` | Deployment      | 1        | `alert_manager` | Carries alert evaluation + scheduled reports (PNG/PDF needs chromium, annotated `molesignal.io/scheduled-reports-renderer: requires-chromium`); ClusterIP                    |
    | `80-web.yaml`           | Deployment      | 2        | —               | `molesignal-web:dev` nginx, `MS_BACKEND=router:5080`, 8080→Service 80; includes an Ingress (host `molesignal.local`)                                                         |
    | `95-ingress.yaml`       | Ingress         | —        | —               | `molesignal-router`, host `molesignal.local`, disables buffering for `/api/v1/query/stream`                                                                                  |

    <Warning>
      `connector` is not a valid `[node].roles` value. Connector runners are owned by
      `alert_manager` in the current service. Remove the legacy connector process from copied
      manifests instead of starting the binary with `["connector"]`.
    </Warning>

    <Warning>
      Both `80-web.yaml` and `95-ingress.yaml` define an Ingress for host `molesignal.local`: one points to the web container, and one points directly to the router. These manifests represent alternative topologies through the web layer or directly to the router. Deploying both causes a conflict, so select one. The router backend in `95-ingress.yaml` references port 80, while the Service in `30-router.yaml` explicitly defines only 5080 and 5082. Align the Service port or Ingress backend before production use.
    </Warning>
  </Tab>
</Tabs>

## Scaling and high availability

<CardGroup cols={2}>
  <Card title="Router — scales horizontally" icon="arrows-left-right">
    Stateless; scale replicas with entry traffic. The rate limiter is in-process and ephemeral: under multiple replicas each replica counts independently, so the effective org QPS ceiling is roughly `configured value × replica count`. When needed, move rate limiting up to a unified gateway or lower the per-replica threshold.
  </Card>

  <Card title="Querier — scales horizontally" icon="magnifying-glass">
    Stateless. A querier process serves the scan RPC over gRPC; distributed scans only trigger with ≥2 querier peers, otherwise the coordinator falls back to the in-process engine with no network hop. The bottleneck is columnar-file reads + query-engine memory.
  </Card>

  <Card title="Intake — stateful" icon="database">
    Use a StatefulSet + a per-replica PVC for a persistent WAL. The Router places via consistent hashing over `org|stream`; scaling changes the mod result and causes rebalancing, so ensure the WAL is flushed before scaling down. Leave enough of a replay window for the readiness probe (the manifest sets a 10s initial delay).
  </Card>

  <Card title="Compactor / AlertManager — single instance" icon="gauge">
    Both are recommended at a single replica: multiple Compactor instances would produce merge conflicts (a lease-table lock is planned for the future), and AlertManager's evaluation / incident state is in-process. Reliability comes from fast restarts rather than multiple replicas.
  </Card>
</CardGroup>

Recommended metrics to monitor: for the write path, watch `wal_append_lock_wait_seconds`, `wal_append_inflight`, `wal_fsync_errors_total`, `file_meta_dump_*`; for the object store, `object_store_operations_total`, `object_store_errors_total`, `object_store_op_duration_seconds`, `object_store_probe_*`; for queries, the cache hit rate `cache_*` and `tantivy_pruned_files_total`; for alerts, `alert_rule_eval_timeout_total`. Combine `/api/v1/healthz`, `/api/v1/readyz`, and the `cluster_nodes` table to observe member liveness.

## Minimal production checklist / validation checklist

<Steps>
  <Step title="External dependencies ready">
    Postgres reachable, `[store.meta].backend = "postgres"` with a correct `dsn`; choose an object store backend of `s3`/`azure`/`gcs` (not `local`), with credentials injected in the precedence order "env vars > credential file > inline".
  </Step>

  <Step title="Roles and interconnect addresses">
    Each process has an explicit `MS_NODE.ROLES`; set `MS_CLUSTER.ADVERTISE_ADDR` to a peer-reachable `host:5082` (K8s uses `$(POD_IP):5082`). A process may carry several roles at once, such as `["intake","querier"]`. The process starts the deduplicated server set and registers under every configured role.
  </Step>

  <Step title="Intake persistence">
    Run the Intake as a StatefulSet + PVC mounting the WAL directory; set `[wal].flush_strategy` / `sync_level` according to durability requirements; leave enough replay delay for the readiness probe.
  </Step>

  <Step title="Single-instance roles">
    Keep Compactor and AlertManager at 1 replica each. Confirm `[compactor].interval_secs`, `retention_days`, and `[storage.file_meta_dump].enabled` match expectations.
  </Step>

  <Step title="Entry point and rate limiting">
    Put an LB in front of the Router; set `[router.rate_limit].intake_qps` / `query_qps` per org; note that rate limiting is approximate under multiple replicas. Disable buffering for `/api/v1/query/stream` on the Ingress.
  </Step>

  <Step title="Observability and probes">
    Wire `/metrics` into Prometheus; point K8s probes at `/api/v1/healthz` (liveness) and `/api/v1/readyz` (readiness). Confirm the object store startup probe passes (otherwise the process will not start).
  </Step>

  <Step title="Security and license">
    Set `MS_AUTH_JWT_SECRET_OVERRIDE` when a fixed JWT secret is required; otherwise the database bootstraps the secret automatically. Inject a cipher key and avoid the development all-zero value in production. Federated query requires the `federated_search` license feature. TLS and automatic certificate issuance or renewal are available in every build without a feature flag and activate at runtime through `[http.tls].enabled`.
  </Step>
</Steps>

<Warning>
  Before production use, note the current boundaries: the distributed-consensus WAL term source is still static because multi-node consensus is not implemented; federated query and OIDC/SAML SSO require the corresponding license features; and changing `[node].roles` requires a process restart.
</Warning>