Note that precision in this context is defined as: > The proportion of events detected that were significant. Precision is 100% if every alert corresponds to a significant event. Note that alerting can become particularly sensitive to nonsignificant events during low-traffic periods (discussed in Low-Traffic Services and Error Budget Alerting). -- from https://sre.google/workbook/alerting-on-slos/ ---------------------------------------------------------------- At present, our Staging environment has the same service-level monitoring rules as those used in Production. Unfortunately, both the recall and precision of these alerts is very poor compared to Production. Much of the problem here is down to the surges in traffic and ensuring low-traffic periods. During these low-traffic periods, the SLO alerts have a very poor sample rate, leading to poor signal. There is an ongoing effort (see https://gitlab.com/gitlab-org/gitlab/-/issues/349074) to improve the base-load traffic in the Staging environment. This relies on CMBR (https://gitlab.com/gitlab-com/gl-infra/cmbr/), a web-crawler that crawls our staging environment. Once we have improved the base-load traffic, we should analyse the alerts we are obtaining from the staging environment and determine the next course of action to improve the precision of the alerts. This may involve one or more of the following actions: 1. Correctly scale the staging environment to handle the traffic it is serving. 1. Fix broken-windows in the staging environment. For example, some endpoints consistently return 500s in staging while working correctly in Production. 1. Adjust SLO thresholds in Staging as compared to Production. 1. Tune the base-load traffic and benchmarking tools. 1. Other actions. Ultimately, the goal here is to improve the quality of staging SLO alerts to the point that we can use the signal of an SLO failure in staging to halt an ongoing deployment. At present, this is only done on Canary environments, not staging. By enabling this in Staging, we may be able to reduce the affect of bad deploys on (a subset of canary) users while also failing-faster. Once we've improved the precision of the staging alerts, the next steps will be: 1. Improve the recall of staging alerts. Recall is defined as "The proportion of significant events detected. Recall is 100% if every significant event results in an alert.". 1. When bad deploys go through staging without detection from our SLO alerts, we'll need to retrospectively assess what (if anything) we could do better to improve the alerting and catch these events. Using the existing corrective-action, infradev and incident review processes would help with this. 1. Improve the processes around SLO alerts. 1. How do we respond to alerts? Is this the duty of the SRE Oncall, or somebody else. 1. Can we automatically shut the gate on new deployments while SLO alerts are firing in Staging? @nolith has already setup the ground-work for this with the Deployment Health indicators, but as far as I'm aware, this doesn't currently cover staging. ---------------------------------------------------- What's been done so far: 1. CMBR is generating better quality base-load traffic (thanks @niskhakova!). This effort is continuing through https://gitlab.com/gitlab-org/gitlab/-/issues/349074 1. A subset of SLO alerts is now being sent to the [`#feed_alerts_staging`](https://gitlab.slack.com/archives/C029L5NMHH8) channel (thanks @reprazent!) 1. [`#feed_alerts_staging`](https://gitlab.slack.com/archives/C029L5NMHH8) is using Slackline for easier-to-read alerts, thanks to https://gitlab.com/gitlab-com/gl-infra/slackline/-/merge_requests/39. 1. With the introduction of Slackline, we are also logging alert data to ELK in the [`alerts_gstg`](https://nonprod-log.gitlab.net/goto/a15ff7f0-7cec-11ec-aec7-f387696fcb42) index. This will give us some information to help with analysis. cc @niskhakova @amyphillips @marin @jarv