fix(alerts): stop paging for grafana lb
fix(alerts): stop paging for grafana lb
What
- Lower the severity of
grafana_google_lb
tos3
to stop paging the on-call
Why
This SLI fires from time to time, and it's mostly unactionable, it's either because a user keeps running bad queries, or an underlying infrastructure issue is happening (which usually pages). Everyone I've talked to gets a bit annoyed by this SLI.
Over the past month, this has paged over 16 times with little action from the on-call to recover.
Setting it to s3
will not page the on-call but we'll still monitor the service which can be helpful to debug problems if want to, if we don't find it helpful we can remove the SLI alltogether.