Self-observability

Process logs stay in the configured console or file output. MoleSignal writes platform metrics, traces, and profiles into the protected _sys organization. Each stored signal uses the exact stream name _molesignal with a corresponding stream type.

Signal	Typed stream	Content
Metrics	`metrics/_molesignal`	Counters, gauges, histograms, and summaries.
Traces	`traces/_molesignal`	Canonical completed spans after tail sampling.
Profiles	`profiles/_molesignal`	Profile metadata linked to object-store pprof blobs.

Only system scope with sys.telemetry.read can query these streams.

Enable self-telemetry collection

[telemetry.self_collect]
enabled = true
metrics_enabled = true
metrics_interval_secs = 15
queue_capacity = 8192
batch_max_events = 256
batch_max_delay_ms = 1000
flush_timeout_secs = 5
profile_kinds = ["cpu"]
profile_interval_secs = 600
profile_duration_secs = 10

The destination cannot be changed to a tenant organization. Public intake, stream mutation, and pipelines cannot create, overwrite, or delete _molesignal. The streams are created with a retention of 7 days. Change it, and the downsampling of the metrics stream, on the stream’s Retention tab in the _sys organization; a restart keeps the values. In split-role deployments, non-intake nodes forward self-telemetry to a healthy Intake node. Set the same MS_SELF_TELEMETRY_CLUSTER_TOKEN on every role.

Backend distributed tracing

MoleSignal instruments HTTP, gRPC, Flight SQL, database, object storage, intake, query, background workers, and Agent stages. Inbound W3C context is validated or replaced. External baggage does not authorize an organization. Sensitive request bodies, SQL values, credentials, prompts, model output, and tool arguments are excluded from spans. The tail sampler retains traces in this order:

deployment or runtime disable;
trusted debug force;
errors;
slow traces;
ordered policy rules;
deterministic normal sampling ratio.

The sampler and self-intake/export queues are bounded and fail open for business traffic. Inspect fixed-cardinality component metrics on /metrics and detailed health at GET /api/v1/system/telemetry.

Node-local pprof

The diagnostic listener is disabled by default:

[profiling]
enabled = true
bind = "127.0.0.1"
port = 5084
allow_remote = false

Capture profiles:

curl --fail --output cpu.pb.gz \
  'http://127.0.0.1:5084/debug/pprof/profile?seconds=30'

curl --fail --output heap.pb.gz \
  'http://127.0.0.1:5084/debug/pprof/heap'

Keep the listener on loopback and use an SSH tunnel. Remote exposure requires allow_remote = true, network controls, TLS in front of the listener, and an administrative bearer credential.

Health and rollout

/api/v1/healthz is public and reports liveness and dependency health. Use it for container and load balancer probes.
/api/v1/readyz reports readiness for traffic after the WAL replay. It requires a bearer token.
/metrics exposes Prometheus metrics without tenant labels.
X-Request-Id and X-Trace-Id correlate requests with retained traces.

For a new distributed-tracing rollout, start with telemetry.trace.force_disabled = true, validate system resources and privacy, enable a canary, then expand the persisted runtime policy.

Configuration

Review every telemetry, tracing, exporter, and profiling setting.

Get started

Data

Observe

Query and correlate

Reliability

AI ops

Collaboration

Admin

Deploy & operate

Best practices

Resources

Enable self-telemetry collection

Backend distributed tracing

Node-local pprof

Health and rollout

Configuration

​Enable self-telemetry collection

​Backend distributed tracing

​Node-local pprof

​Health and rollout

Configuration

Enable self-telemetry collection

Backend distributed tracing

Node-local pprof

Health and rollout