Route SLO alerts to pagerduty
gitlab-com/runbooks!1344 (merged) is routing latency SLO alerts to pagerduty as they seem to be very accurate in indicating real production issues.
We also should send other SLO alerts in https://gitlab.com/gitlab-com/runbooks/blob/master/rules/general-service-alerts.yml to pagerduty after we trimmed down the false alert rate (https://gitlab.com/gitlab-org/gitlab-ce/issues/66166):
-
error ratios alerts -
saturation alerts -
service availability alerts
Operation rate alerts probably shouldn't go to pagerduty, as they will manifest in higher latencies or error ratios when they affect production.
Edited by Henri Philipps