Skip to main content
Creates an alert rule. The call requires alerts.manage.

​
How a rule evaluates

The alert role evaluates every enabled scheduled and anomaly rule every [alert].eval_interval_secs, 30 seconds by default. A real_time rule is matched on the intake path and opens its incident as records arrive.
  1. The evaluator runs query.statement over the window from now - period_secs to now. A sql statement runs in the SQL engine. A promql statement runs in the PromQL engine as an instant query at now. An APM rule reads its figure from the APM aggregates of the window instead, and a RUM rule from the RUM data of the window. See APM rules and RUM rules.
  2. The evaluator reads the first column of the first row as a number. A result without a number counts as no data; trigger.no_data decides what that does, see Recovery and missing data. A promql result lists the sample value first, so the first series decides. Aggregate the expression to one series, for example with sum(...), or split the rule by labels with query.group_by; see Rules split by columns.
  3. The number is compared with trigger, or with every tier of thresholds when that array is not empty.
  4. After the condition matches on for_periods consecutive evaluations, an incident opens. A rule has at most one active incident at a time, or one for each group when it is split by columns. When silence_secs is above 0, the rule opens no new incident until that many seconds have passed since the previous incident of the rule was created, even when the previous one is already resolved.
  5. When the recovery condition is met, the evaluator resolves the open incident. By default that is the first evaluation the condition stops matching. An acknowledged incident stays acknowledged until a user resolves it.
  6. After every completed evaluation, the evaluator stores the time in last_eval_at and the outcome in last_state. A query that fails or times out leaves both fields unchanged.
For scheduled and anomaly rules, the call does not check that the stream or the statement is valid. Run Test a rule before enabling a rule. A real_time rule is checked against the limits below when it is saved. Body query Columns named service, host, and trace_id in the first row enrich the incident. Such columns add the service and host labels and fill affected_services, host_ids, and trace_ids. trigger An anomaly rule ignores operator and threshold and still uses for_periods. A rule with thresholds ignores operator, threshold, and for_periods of trigger, but the object stays required. thresholds Each tier has severity (info, warning, error, or critical), operator, threshold, and for_periods. A tier counts its own consecutive matches. The highest severity among the tiers that fired sets the incident severity. When an open incident rises to a higher severity, escalation restarts at the first step. Without tiers, the severity of an incident is the rule severity, then a severity label, then warning. For an anomaly rule without either, a deviation of at least 10 times the baseline gives critical, and at least 3 times gives error. anomaly_params

​
Recovery and missing data

trigger.recovery keeps an incident open until the value is clearly back. A value that hovers around the threshold otherwise fires and recovers over and over. trigger.no_data decides what an evaluation without a value counts as: the query returned no row or no number, or a group of a rule split by columns is missing from the result. A no-data incident has its own fingerprint, so it can be open beside an incident the threshold opened. It carries the label alert.condition = no_data and the summary of the rule or group, and notifications title it NO DATA · <summary>. Its severity is the rule’s severity, then a severity label, then warning. silence_secs applies to it as well. When someone resolves it while the value is still missing, it is not reopened before data returns, and a group of a split rule in that case is no longer followed. A real_time rule takes neither recovery nor no_data.

​
Rules split by columns

A scheduled rule with query.group_by evaluates every group of its result on its own. A group is the rows that have the same values in those columns, matched to the columns of the result without regard to case. An APM, a RUM, a real_time, or an anomaly rule with group_by answers 400.
  1. A sql statement runs with a limit of 101 rows. A promql statement returns every series. Return one row for each group, the worst first.
  2. The first row of a group gives its value: the first numeric column other than the named columns and _timestamp, ts, time, or timestamp.
  3. The value is compared with trigger or thresholds, and the group counts its own for_periods.
  4. A group that fires opens its own incident. The incident gets the values of the group as labels, and the rule name followed by the values as its summary, for example Errors per host · web-1. silence_secs applies to every group on its own.
  5. A group that meets the recovery condition has its open incident resolved. A group that was in an earlier result and is not in this one has no value this evaluation, and trigger.no_data decides what that does for it.
  6. last_state is firing while any group fires, no_data while any group has no value and keeps its state, pending while any group matches, and healthy otherwise.
A round reads at most 100 groups. When the result has more, the rest are not evaluated, and in that round a group missing from the result keeps its incident. When the result has rows but no column of a name in group_by, the round fails and changes nothing. Test a rule reports every group.

​
APM rules

A rule with query.apm watches a figure of an APM service, or of one of its endpoints, read from the same aggregates as the APM pages. An APM rule must be scheduled. The call stores query.stream as null and, when query.statement is empty, fills it with a description such as apm:p95 checkout POST /orders. query.language is stored as sent and has no effect. The window reads minute aggregates up to 24 hours and hourly aggregates beyond that. Without requests in the window, request_count, error_count, new_error_groups, and throughput read 0, and the other metrics have no value, so no threshold matches. The result has the columns value, service, and trace_id, so the incident gets the service label, the service in affected_services, and a sample trace ID.

​
RUM rules

A rule with query.rum watches a figure of a RUM application, read the way the RUM pages read it. A RUM rule is scheduled or anomaly, and a rule names an APM figure or a RUM figure, not both. The call stores query.stream as null and, when query.statement is empty, fills it with a description such as rum:lcp_p75 checkout-web /checkout. query.language is stored as sent and has no effect. The application has to exist, or the call answers 400. metric is one of session_count, view_count, error_count, error_rate, crash_free_sessions, frustration_rate, failed_request_rate, lcp_p75, inp_p75, cls_p75, ttfb_p75, new_issues, and regressed_issues. What each one is, its unit, and the rows it needs are in RUM alerts. An anomaly rule takes every figure but new_issues and regressed_issues. period_secs is from 60 seconds to 6 hours. A window with no rows reads 0 for a count, and no value for the other figures, so no threshold matches. The result has the columns value and application in one row. When an incident opens, the rule also looks for up to three sample sessions and keeps them with the incident. See RUM alerts.
Errors See Alerting and incident response for the console workflow.
Last modified on October 26, 2026