Measure first
Find the streams that carry the volume before changing anything.
Write down the numbers of the largest streams. They are the baseline to judge each change against.
Drop what nobody reads before it is stored
The cheapest byte is the one that is never stored. For logs, a realtime pipeline runs during intake, before storage:- Filter events drops health checks, readiness probes, and debug lines that no one reads in production.
- Sample keeps a share of a noisy but useful event type. With Sample by field, the events that share a value, such as one request ID, are kept or dropped together.
- Remove fields drops large fields that no query uses, such as full request bodies.
info in production, or a collector processor
that drops noise, saves the network as well. Dropped events are gone for good; sample only what is
repeated, never errors.
Trace volume is best reduced at the source or in the OpenTelemetry Collector. Keep every span of a
trace together, and remember that APM figures and SLOs count only the spans that arrive. For RUM, set
session sampling per application.
Keep data only as long as it is used
Set retention from how long investigations really look back, and leave margin for slow discoveries.
Shortening retention deletes the data that is then past it in the next cleanup, and downsampling
deletes the original samples: both cannot be undone. Keep Merge small files by day on for logs and
traces, so queries over many days open fewer files.
Keep metric series bounded
Each distinct combination of label values is a series. Labels with unbounded values, such as user IDs, request IDs, or raw URLs, create series without limit, and the cardinality limits then refuse new series for the whole organization.- Check Cardinality on Metrics after adding a label.
- Replace raw paths with route templates, and IDs with classes such as plan or region.
- Put per-request detail in logs or spans, which cost nothing per value.
Keep queries cheap
Acceptance criteria
- The five largest streams are known, with their daily volume before and after the changes.
- No stream stores health checks, readiness probes, or production debug logs by accident.
- Each stream has a retention set on purpose, and long-lived metrics are downsampled.
- Metric series counts are stable from week to week and far below the limits.
- Large streams have a maximum query range, and their common filters are indexed.