Agree on the questions first
Choose a small set of operational questions: whether requests succeed, where latency accumulates, and which dependency fails. Associate each question with a signal and a responsible team.
Use stable service and environment values across signals. Keep request identifiers in logs and traces; avoid using a unique request or customer ID as a metric label because each value combination adds a series.
Start with a controlled cohort
Use a test environment and synthetic customer data first. Send a successful request, a dependency failure, and a slow request through the same path. Verify that timestamps, identifiers, and field types support investigation across signals. Connect each source with the guide on Datasource: it fills in the endpoint and the token, and Verify data is arriving tells whether data arrived in the last hour. Then enable collection for a small production cohort. Expand after reviewing rejected events, ingestion volume, query latency, and coverage of failures. Define the observation period and acceptance limits before rollout; a quiet interval does not exercise a peak-traffic workload.Choose stream boundaries deliberately
Separate log streams when retention, schema, or access requirements differ. Prefer fields for service, environment, and release when those requirements are shared. Avoid creating a stream for every instance or deployment version. See Design streams and fields that stay useful. Keep a field’s type stable across producers. MoleSignal rejects incompatible field values with a type mismatch. Test transformations against missing fields, unexpected types, and large messages before attaching a pipeline to production traffic. Remove secrets at the producer or collector before sending data. Preservetrace_id through transformations. Use a new version field to distinguish releases rather than changing field meanings in place.
Balance retention and investigation coverage
Set retention according to the time needed to discover and investigate failures. Keep high-volume diagnostic logs for a shorter period only when the incident response process can work within that window. Retention cleanup cannot be undone by later increasing the retention period. See Control telemetry volume and cost. When reducing trace collection, verify that logs still retain useful error context. A missing trace may reflect collection or sampling gaps; do not interpret it as proof that the request succeeded.Acceptance criteria
- A known failed request can be found in logs and correlated with its captured trace.
- Test events preserve types and identifiers after pipeline processing.
- A representative traffic interval stays within the agreed volume and query-latency limits.
- Sensitive test markers are absent from received telemetry.
- The owning team has a rollback plan for collection and transformation changes.