> ## Documentation Index
> Fetch the complete documentation index at: https://docs.molesignal.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Self-observability

> Operate MoleSignal's own logs, metrics, traces, profiles, health, and node-local diagnostics.

Process logs stay in the configured console or file output. MoleSignal writes platform metrics,
traces, and profiles into the protected `_sys` organization. Each stored signal uses the exact
stream name `_molesignal` with a corresponding stream type.

| Signal   | Typed stream           | Content                                              |
| -------- | ---------------------- | ---------------------------------------------------- |
| Metrics  | `metrics/_molesignal`  | Counters, gauges, histograms, and summaries.         |
| Traces   | `traces/_molesignal`   | Canonical completed spans after tail sampling.       |
| Profiles | `profiles/_molesignal` | Profile metadata linked to object-store pprof blobs. |

Only system scope with `sys.telemetry.read` can query these streams.

## Enable self-telemetry collection

```toml theme={null}
[telemetry.self_collect]
enabled = true
retention_days = 7
metrics_retention_days = 7
traces_retention_days = 7
profiles_retention_days = 7
metrics_enabled = true
metrics_interval_secs = 15
queue_capacity = 8192
batch_max_events = 256
batch_max_delay_ms = 1000
flush_timeout_secs = 5
profile_kinds = ["cpu"]
profile_interval_secs = 600
profile_duration_secs = 10
```

The destination cannot be changed to a tenant organization. Public intake, stream mutation, and
pipelines cannot create, overwrite, or delete `_molesignal`.

In split-role deployments, non-intake nodes forward self-telemetry to a healthy Intake node. Set the
same `MS_SELF_TELEMETRY_CLUSTER_TOKEN` on every role.

## Backend distributed tracing

MoleSignal instruments HTTP, gRPC, Flight SQL, database, object storage, intake, query,
background workers, and Agent stages.

Inbound W3C context is validated or replaced. External baggage does not authorize an organization.
Sensitive request bodies, SQL values, credentials, prompts, model output, and tool arguments are
excluded from spans.

The tail sampler retains traces in this order:

1. deployment or runtime disable;
2. trusted debug force;
3. errors;
4. slow traces;
5. ordered policy rules;
6. deterministic normal sampling ratio.

The sampler and self-intake/export queues are bounded and fail open for business traffic. Inspect
fixed-cardinality component metrics on `/metrics` and detailed health at
`GET /api/v1/system/telemetry`.

## Node-local pprof

The diagnostic listener is disabled by default:

```toml theme={null}
[profiling]
enabled = true
bind = "127.0.0.1"
port = 5084
allow_remote = false
```

Capture profiles:

```bash theme={null}
curl --fail --output cpu.pb.gz \
  'http://127.0.0.1:5084/debug/pprof/profile?seconds=30'

curl --fail --output heap.pb.gz \
  'http://127.0.0.1:5084/debug/pprof/heap'
```

Keep the listener on loopback and use an SSH tunnel. Remote exposure requires
`allow_remote = true`, network controls, TLS in front of the listener, and an administrative
bearer credential.

## Health and rollout

* `/api/v1/healthz` reports process liveness.
* `/api/v1/readyz` reports readiness for traffic.
* `/metrics` exposes Prometheus metrics without tenant labels.
* `X-Request-Id` and `X-Trace-Id` correlate requests with retained traces.

For a new distributed-tracing rollout, start with `telemetry.trace.force_disabled = true`, validate
system resources and privacy, enable a canary, then expand the persisted runtime policy.

<Card title="Configuration" icon="sliders" href="/en-US/configuration">
  Review every telemetry, tracing, exporter, and profiling setting.
</Card>