Skip to content

fix(alerts): stop paging for grafana lb

Steve Xuereb requested to merge fix/grafana-lb into master

fix(alerts): stop paging for grafana lb

What

  • Lower the severity of grafana_google_lb to s3 to stop paging the on-call

Why

This SLI fires from time to time, and it's mostly unactionable, it's either because a user keeps running bad queries, or an underlying infrastructure issue is happening (which usually pages). Everyone I've talked to gets a bit annoyed by this SLI.

Over the past month, this has paged over 16 times with little action from the on-call to recover.

Screenshot_2023-01-16_at_13.51.38

source

Setting it to s3 will not page the on-call but we'll still monitor the service which can be helpful to debug problems if want to, if we don't find it helpful we can remove the SLI alltogether.

Merge request reports