Skip to main content
A scheduled pipeline reads a source stream over a time window, applies an ordered chain of VRL transforms, and writes the output to a target stream (standard intake) — optionally fanning the same events out to external connectors (S3, Kafka). Pipelines run on a simplified interval schedule, or on demand as a backfill over a historical window.
Scheduled pipelines transform data after intake, as a recurring job. To transform events on the intake hot path instead, attach a function to the intake step.
Scheduled pipeline

Scheduled pipeline

​
Build a pipeline in the graph editor

Open Pipelines → New to lay out the flow on a canvas: source → transform → sink. Nodes are draggable, handles support connections, and the editor validates the graph during layout (missing source/sink, a transform without a name or script, duplicate stream names, a source that also appears as a sink). The first source and first stream sink are saved as the pipeline’s source_stream and target_stream.

​
Transforms

Each transform is a VRL script. The transform inspector supports reuse of a built-in preset or any saved function from the reuse picker. Selecting an entry fills the step script, which remains editable inline. Built-in presets ship with every instance and are read-only:
Presets and saved functions share one catalog, with both types on the Functions page. Built-in presets are badged and read-only; use Copy script to create a custom function.

​
Extend tables

An extend table is a key → record lookup table available to a transform. Open Extend tables, create a table, add rows (a key plus named fields), and perform the lookup from VRL:
Lookups happen in memory and add no query cost to the run.

​
Sinks and connector egress

A sink is either a target stream (the default — events are written through standard intake and become queryable) or an external connector: Add connectors under Pipelines → Connectors, then select connectors as sinks in the graph editor. A pipeline can write to a stream and egress to connectors in the same run.

​
Hide the source stream from queries

When fanning a catch-all source stream out into per-service or per-tenant target streams — for example with the route-by-service preset above — the source stream keeps accumulating the raw, unrouted events. To keep queries and dashboards pointed only at the routed streams, mark the source stream non-queryable: open Streams → source → Settings and turn Queryable off. A non-queryable stream still accepts and retains data and feeds the pipeline, but the stream is hidden from the query stream selectors and rejected by SQL and PromQL search with stream is not queryable. Flip the toggle back on at any time.

​
Schedule and lookback

The runner does not parse standard cron expressions. A schedule outside the supported every: syntax is skipped. On each tick the runner reads [now - lookback_secs, now] from the source, applies the chain, and writes the output. Every run is recorded — view history under the pipeline’s Runs tab, or via GET /api/v1/scheduled_pipelines/{id}/runs (state, scanned_rows, error).
The schedule runner is a singleton and runs only on the alert-manager (or standalone) node, so enabled pipelines fire once per interval rather than once per node.

​
Backfill

To process a historical window on demand, submit a backfill:
The window must be ≤ 31 days. The request returns 202 with a job_id and a monitor URL; the backfill runs through the async search-job worker — read the source window, apply the same transform chain, write the target stream and egress.

​
Permissions

  • pipelines.read lists pipelines and reads pipeline run history.
  • pipelines.create creates a pipeline.
  • pipelines.edit changes the pipeline graph, transforms, schedule, or lookback.
  • pipelines.pause enables or pauses scheduled execution.
  • pipelines.run submits a backfill.
  • pipelines.delete deletes a pipeline.

Pipeline API

Create, update, list, and backfill scheduled pipelines over the HTTP API.
Last modified on August 9, 2026