### Problem On 2026-04-11, GitLab.com experienced a brief but significant outage ([INC-9131](https://app.incident.io/gitlab/incidents/9131)) where Puma web workers became unresponsive fleet-wide, causing Workhorse to return HTTP 502 errors. Multiple alerting signals fired during the incident: - Web service loadbalancer SLI error rate hit 2.008%, violating the SLO - API Workhorse SLI error rate hit 6.092% on canary stage, violating the SLO - Blackbox probe availability for `/explore` & `/users/sign_in` - 558,870 total HTTP 5xx responses across many routes in the 01:00–01:20 UTC window The SRE on-call responded to these signals. However, because the impact was brief and self-resolving, the incident was downgraded to S3. The on-call engineer did not have access to accumulated SLA downtime minute information to prevent this downgrade. Because our incident review process is gated on S1/S2 severity, no incident review was triggered. What we didn't know at the time was that this incident, combined with other smaller events throughout the day, contributed to a significant accumulation of SLA downtime minutes — visible as a spike on April 11 in the [April SLA data](https://fviegas-64d3a6.gitlab.io/timeline_april_2026_final.html) and [SLA reporting](https://gitlab.com/groups/gitlab-operating-model/-/work_items/339#note_3314179900). We cannot explain to stakeholders what the root cause was or what corrective actions are being taken, because none of this was investigated. The web overview dashboard shows the SLI violations during the incident window:  [Web overview dashboard](https://dashboards.gitlab.net/d/web-main/web3a-overview?orgId=1&from=2026-04-11T00:00:00.000Z&to=2026-04-11T02:38:07.740Z&timezone=utc&var-PROMETHEUS_DS=mimir-gitlab-gprd&var-environment=gprd&var-stage=main) The Workhorse alert analysis shows the alerts that fired:  [Workhorse alert analysis](https://dashboards.gitlab.net/goto/cflc9f77306iof?orgId=1) Beyond the initial incident, the slow burn of downtime minutes throughout the rest of the day was visible in Cloudflare metrics but never identified as a single correlated event:  [Grafana explore query](https://dashboards.gitlab.net/explore?schemaVersion=1&panes=%7B%22a0b%22:%7B%22datasource%22:%22mimir-gitlab-ops%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22clamp_max(rate(cloudflare_zone_requests_status%7Bzone%3D%5C%22gitlab.com%5C%22,%20status%3D~%5C%225..%5C%22%7D%5B5m%5D),%20200)%22,%22range%22:true,%22instant%22:true,%22datasource%22:%7B%22type%22:%22prometheus%22,%22uid%22:%22mimir-gitlab-ops%22%7D,%22editorMode%22:%22code%22,%22legendFormat%22:%22__auto%22%7D%5D,%22range%22:%7B%22from%22:%221775865600000%22,%22to%22:%221775951999000%22%7D,%22compact%22:false%7D%7D&orgId=1) showing `rate(cloudflare_zone_requests_status{zone="gitlab.com", status=~"5.."}[5m])` on 2026-04-11. ### Corrective actions #### 1. Revamp the Cloudflare service SLIs in the metrics catalog The current [Cloudflare service definition](https://gitlab.com/gitlab-com/runbooks/-/blob/master/metrics-catalog/services/cloudflare.jsonnet) has several issues that reduce its effectiveness as an alerting signal: - **High error ratio threshold:** The `monitoringThresholds.errorRatio` is set to `0.99` (99%), with a [comment](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/5465) noting monitoring data may be unreliable. This threshold is too permissive to catch the kind of error spikes we saw on April 11. - **Non-paging severity:** All three SLIs (`gitlab_zone`, `gitlab_net_zone`, `cloud_gitlab_zone`) are configured as `s3`. Since only `s1` and `s2` severities trigger PagerDuty pages (per `libsonnet/slo-alerts/service-level-alerts.libsonnet`), Cloudflare SLI alerts only go to Slack and are easily missed. At least `gitlab_zone` should be able to page the oncall. - **502 exclusion on `cloud_gitlab_zone`:** The `cloud_gitlab_zone` SLI explicitly excludes HTTP 502 responses from its error rate calculation (`{ ne: '502' }`). On 2026-04-11, 502s were the dominant error type — 534,977 out of 551,881 5xx responses were HTTP 502. **Proposed changes:** - Lower the `errorRatio` threshold to a value that reflects actual SLA targets. - Raise the severity of at least the `gitlab_zone` SLI to `s2` (minimum) so it becomes a paging alert. - Remove the `{ ne: '502' }` exclusion from the `cloud_gitlab_zone` SLI error rate selector. Before doing so, we need to understand the signal quality: were 502s excluded because they are genuinely noisy, or were they masking a real underlying problem? On 2026-04-11, the 502s on `gitlab.com` were caused by Workhorse being unable to connect to Puma at all. If re-including 502s causes the alert to fire frequently, that should be treated as a problem to investigate and fix at the source — not a reason to re-exclude them from the SLI. #### 2. Enforce incident review for incidents that contribute to SLA downtime minutes Currently, SREs don't have any visibility into the SLA during or after an incident. There is planned work for SLA breach dashboards that is somewhat stalled: https://gitlab.com/gitlab-com/gl-infra/observability/team/-/work_items/4543 We can't have realtime SLA information (the data comes from log processing), but we can build automation that runs after the fact: - Both incident data and SLA downtime minute data are available in BigQuery from the [SQLMesh catalog](https://gitlab.com/gitlab-com/gl-infra/data/sqlmesh-catalog). - Automation could correlate incidents with SLA downtime minutes, add SLA impact information to the incident in incident.io, and trigger an incident review if one was skipped due to the incident being classified at too low a severity. This ensures that any incident contributing to SLA downtime gets a proper root cause analysis, regardless of the severity assigned in the moment. #### 3. Notify on SLA downtime minutes when they become available We can't use regular alerting for SLA breaches because the data doesn't arrive in realtime. But we should notify the SRE team to investigate when accumulated downtime minutes increase — especially if there is no correlated incident that explains the downtime. This is a process change with an open question: **should we declare an incident after the fact to trigger the incident review process, or should we handle this differently?** Declaring a retroactive incident has the advantage of reusing existing incident review workflows, but may feel awkward if the issue has already self-resolved hours ago. ### References - [April SLA data and discussion](https://gitlab.com/groups/gitlab-operating-model/-/work_items/339#note_3314179900) - [April SLA timeline](https://fviegas-64d3a6.gitlab.io/timeline_april_2026_final.html) - [INC-9131](https://app.incident.io/gitlab/incidents/9131) — the 2026-04-11 incident - [Cloudflare service definition](https://gitlab.com/gitlab-com/runbooks/-/blob/master/metrics-catalog/services/cloudflare.jsonnet) in runbooks - [Dashboard for internal SLA breach reporting](https://gitlab.com/gitlab-com/gl-infra/observability/team/-/work_items/4543) — stalled work on SLA visibility - Prior similar incidents: INC-8093, INC-8100, INC-8989 — all transient self-resolving incidents that were auto-declined with no root cause identified