Self-observability

Process logs stay in the configured console or file output. MoleSignal writes platform metrics, traces, and profiles into the protected _sys organization. Each stored signal uses the exact stream name _molesignal with a corresponding stream type.

Signal	Typed stream	Content
Metrics	`metrics/_molesignal`	Counters, gauges, histograms, and summaries.
Traces	`traces/_molesignal`	Canonical completed spans after tail sampling.
Profiles	`profiles/_molesignal`	Profile metadata linked to object-store pprof blobs.

Only system scope with sys.telemetry.read can query these streams.

Enable self-telemetry collection

[telemetry.self_collect]
enabled = true
retention_days = 7
metrics_retention_days = 7
traces_retention_days = 7
profiles_retention_days = 7
metrics_enabled = true
metrics_interval_secs = 15
queue_capacity = 8192
batch_max_events = 256
batch_max_delay_ms = 1000
flush_timeout_secs = 5
profile_kinds = ["cpu"]
profile_interval_secs = 600
profile_duration_secs = 10

The destination cannot be changed to a tenant organization. Public intake, stream mutation, and pipelines cannot create, overwrite, or delete _molesignal. In split-role deployments, non-intake nodes forward self-telemetry to a healthy Intake node. Set the same MS_SELF_TELEMETRY_CLUSTER_TOKEN on every role.

Backend distributed tracing

MoleSignal instruments HTTP, gRPC, Flight SQL, database, object storage, intake, query, background workers, and Agent stages. Inbound W3C context is validated or replaced. External baggage does not authorize an organization. Sensitive request bodies, SQL values, credentials, prompts, model output, and tool arguments are excluded from spans. The tail sampler retains traces in this order:

deployment or runtime disable;
trusted debug force;
errors;
slow traces;
ordered policy rules;
deterministic normal sampling ratio.

The sampler and self-intake/export queues are bounded and fail open for business traffic. Inspect fixed-cardinality component metrics on /metrics and detailed health at GET /api/v1/system/telemetry.

Node-local pprof

The diagnostic listener is disabled by default:

[profiling]
enabled = true
bind = "127.0.0.1"
port = 5084
allow_remote = false

Capture profiles:

curl --fail --output cpu.pb.gz \
  'http://127.0.0.1:5084/debug/pprof/profile?seconds=30'

curl --fail --output heap.pb.gz \
  'http://127.0.0.1:5084/debug/pprof/heap'

Keep the listener on loopback and use an SSH tunnel. Remote exposure requires allow_remote = true, network controls, TLS in front of the listener, and an administrative bearer credential.

Health and rollout

/api/v1/healthz reports process liveness.
/api/v1/readyz reports readiness for traffic.
/metrics exposes Prometheus metrics without tenant labels.
X-Request-Id and X-Trace-Id correlate requests with retained traces.

For a new distributed-tracing rollout, start with telemetry.trace.force_disabled = true, validate system resources and privacy, enable a canary, then expand the persisted runtime policy.

Configuration

Review every telemetry, tracing, exporter, and profiling setting.

Getting Started

Observe

Reliability

AI Ops

Data

Collaboration

Admin

Deploy & Operate

Enable self-telemetry collection

Backend distributed tracing

Node-local pprof

Health and rollout

Configuration

​Enable self-telemetry collection

​Backend distributed tracing

​Node-local pprof

​Health and rollout

Configuration

Enable self-telemetry collection

Backend distributed tracing

Node-local pprof

Health and rollout