Alert fatigue is a routing problem, not a volume problem

When we ask new customers what is wrong with their alerts, the first answer is nearly always “there are too many”. When we look at the channel, the number is usually modest, perhaps thirty a day. The real problem is that twenty-eight of them are for somebody else.

The shared channel

A shared channel feels safe. Everyone sees everything, so nothing is missed. In practice each person scans the channel, decides that an alert probably is not theirs, and moves on. Because everyone does this, the alert that is theirs is also left for someone else.

Psychologists have a name for this, and so does every on-call engineer: the bystander effect. It gets stronger as the channel gets bigger.

The fix is not fewer alerts. It is that each alert has one place to go and one person who is expected to act on it.

Three questions every alert should answer

Before an alert is sent, the system should know:

1. Who owns the thing that is wrong? A team, a rota, or a named person.
2. How urgent is it? Does somebody need to act in the next ten minutes, or by tomorrow morning?
3. What is affected? The downstream tables and dashboards, so the owner can decide quickly.

If you can answer those, you can route. If you cannot, the alert is a rumour.

Routing by owner

In Tidewater an alert rule states who receives an alert in terms of ownership instead of a fixed channel. Here is a rule in the configuration format:

rules:
  - name: Late data goes to the owner
    when:
      event: freshness.late
      urgency: [high, normal]
    notify:
      owner: team
      fallback: '#data-platform'

  - name: Low urgency waits for the morning digest
    when:
      event: freshness.late
      urgency: low
    notify:
      owner: team
      mode: digest
      at: '08:30'

The first rule sends high and normal urgency alerts to whichever team owns the late table. The second rule collects low urgency alerts and sends them once, at 08:30 in the owning team’s time zone.

Where owners come from

Most teams already have owners written down somewhere, just not in one place. We read them from three sources, in this order:

1. A meta.owner field in dbt models.
2. A tag or label on the pipeline in your orchestrator.
3. A setting in Tidewater, for anything else.

When a table has no owner, the alert goes to the fallback channel and says so. That sentence, “this table has no owner”, is surprisingly effective. Teams add owners to stop seeing it.

Use a digest for everything that can wait

A late staging table at 02:00 does not need a person. It needs a record. A digest gathers the alerts that can wait into one message, ordered by how much is affected, and sends it when somebody will read it.

What matters is that the rule for “can wait” is explicit. We suggest this test: if nobody acts on it for four hours, does a customer or a decision suffer? If not, it is low urgency.

What changes

Customers who moved from one shared channel to ownership routing tell us the same few things:

• The number of alerts a person sees each day drops by about two thirds, even though the number sent is unchanged.
• Acknowledgement time falls, because the first person to see an alert is the one who should act.
• The shared channel becomes quiet enough to use for conversation again.

None of that required deleting a single alert. It required deciding, once, who each one is for.

Where to start

Pick your five noisiest alert rules. For each, name an owner for the tables involved and decide an urgency. Change the destination, leave the rule as it is, and look at the results after a week. If an alert still gets ignored, it is either routed to the wrong person or not worth sending.