Skip to content

Traffic absent alerts causing pager noise

Summary

We're getting a lot of pages for sparse metrics (metrics that are intermittently reported). When they go missing, we get alerted because traffic seems to be "absent" - but there was never any traffic in the first place.

I suspect it has something to do with how we're using offset 1h or unless on our alerting rules.

Related Incident(s)

Desired Outcome/Acceptance Criteria

We are no longer getting *TrafficAbsent alerts for things that had no traffic to begin with.

Associated Services

Corrective Action Issue Checklist

  • Link the incident(s) this corrective action arose out of
  • Give context for what problem this corrective action is trying to prevent from re-occurring
  • Assign a severity label (this is the highest sev of related incidents, defaults to 'severity::4')
  • Assign a priority (this will default to 'Reliability::P4')
Edited by Adeline Yeung