alerts.inhibitions.read; creating, changing and deleting take alerts.inhibitions.manage.
Inhibition and silences
Both hold back only notifications and escalation. Neither affects the incident itself, status pages or tickets.
Create a rule
Click New inhibit rule:
Label conditions are written as in silence policies: all of them must hit, and
=~ and !~ are regular expressions over the whole value. Besides an incident’s labels, three more are understood:
serviceandhost: when the incident has no such label, its affected service or host;alert.rule_id: the ID of the rule or alert source that opened it, for “rule A holds back rule B”.
Common rules
On one service, the worse holds back the lesser. This is how a new rule starts:- These labels must match:
service - The source must be worse than the target: on
- Source label condition:
service=database - Target label condition:
tier=api - These labels must match:
region(only in the same region) - The source must be worse than the target: off
How it is decided
Escalation dispatch decides each round (10 seconds by default) and writes the result on the incident. To guarantee that an incident held back always has one upstream that really tells someone:- A silenced incident is no source. Nobody hears about it, so holding others back would leave nobody told.
- An acknowledged incident stays a source. Someone is on it.
- An incident merged into another counts on neither side.
- An incident a person opened is never held back, though it can hold others back.
- Holding back chains. When A holds back B and B holds back C, neither B nor C notifies; A does.
- Incidents holding each other back leave one. When two rules say A holds back B and B holds back A, the worse one (the earlier among equals) notifies as usual.
Two buffers in time
The target waits. Symptoms often report before their cause: the error rate crosses its line first, and the database rule evaluates in its next round. When an incident that could be held back opens, its notifications wait for Target waits. A source that arrives in that time holds it back; if none does, it notifies as usual. Set to 0 it still waits 15 seconds: the next dispatch round decides, and the first notification must not run ahead of it. Only incidents that could be held back wait: the most severe ones do not when the source must be worse, nor do those without the labels that must match. Holding on after recovery. When the cause recovers its symptoms soon follow, and nobody needs paging for them. After a source recovers, its targets stay held back this long, and longer if another source arrives. Held notifications go out only for what is still open then.What happens to notifications
On the pages
- Incident list: an incident held back says Inhibited in its summary line, and an Inhibited view appears while there are any.
- Incident page: says which incident and which rule hold it back, with a link to the source; one that is waiting says until when. Notify as usual beside it releases the incident: its held notifications go out now and it is not inhibited again. That takes
alerts.incidents.update, and is recorded in the timeline and the audit log. - Timeline: an entry when it starts being held back and when that ends (source recovered, rule changed or deleted, someone released it).
- Rule list: a rule whose incidents are all held back shows Firing (inhibited).
- Inhibition page: each rule says how many incidents it holds back now. Deleting one that does says they will notify at once.