Deployment Blockers - Week: 2026-04-20-2026-04-26

Overview

Start date End date Production deployments blocked for # of blockers
2026-04-20 2026-04-26 39.0 44

Blocker types overview

Blocker Type Time blocked on gstg Time blocked on gprd
RootCauseExternal-Dependency 1.0 1.0
RootCauseFlaky-Test 4.0 4.0
RootCauseDB-Migration 2.5 2.5
RootCauseSaturation 3.0 3.0
RootCauseSoftware-Change 25.0 25.0
RootCauseConfig-Change 3.5 3.5
Total 39.0 39.0

Weekly overview

Issue Blocker type gstg gprd
2026-04-20: Repeated CI Job failures with Gital... (gitlab-com/gl-infra/production#21828 - closed) RootCauseExternal-Dependency 1.0 1.0
QA failure: (#24936 - closed) 0.0 0.0
2026-04-20: The ci_runner_jobs SLI of the ci-ru... (gitlab-com/gl-infra/production#21835 - closed) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/24941+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/24949+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/24953+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/24954+ 0.0 0.0
QA failure: (#24959 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/24963+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/24964+ 0.0 0.0
QA failure: (#24966 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: (#24972 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/24976+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/24977+ 0.0 0.0
2026-04-22: Post-deploy migrations failed in gprd (gitlab-com/gl-infra/production#21856 - closed) RootCauseDB-Migration 0.5 0.5
QA failures on gstg-cny (#25007 - closed) 0.0 0.0
2026-04-22: Redis cluster down causing 500s and... (gitlab-com/gl-infra/production#21858 - closed) RootCauseSaturation 3.0 3.0
2026-04-22: Multiple smoke failures on staging (gitlab-com/gl-infra/production#21859 - closed) RootCauseSoftware-Change 24.0 24.0
QA failures on gstg-cny (#25028 - closed) 0.0 0.0
QA failures on gstg-cny (#25038 - closed) 0.0 0.0
2026-04-23: PGBouncer client connections per pr... (gitlab-com/gl-infra/production#21864 - closed) RootCauseConfig-Change 0.5 0.5
QA failures on gstg-cny (#25054 - closed) RootCauseFlaky-Test 0.5 0.5
2026-04-23: Requests to ai-assisted service exc... (gitlab-com/gl-infra/production#21867 - closed) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/25064+ 0.0 0.0
QA failures on gstg-cny (#25065 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/25066+ 0.0 0.0
2026-04-23: Prod PDM fails due to lock timeout (gitlab-com/gl-infra/production#21872 - closed) RootCauseDB-Migration 2.0 2.0
2026-04-23: Pages don't load due to a JS bundle... (gitlab-com/gl-infra/production#21873 - closed) RootCauseConfig-Change 3.0 3.0
QA failure: (#25093 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/25096+ 0.0 0.0
2026-04-23: Redis shared state primary CPU util... (gitlab-com/gl-infra/production#21876 - closed) RootCauseSoftware-Change 1.0 1.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/25122+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/25144+ 0.0 0.0
QA failure: (#25148 - closed) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/25150+ 0.0 0.0
QA failure: qa/specs/features/ee/browser_ui/2_p... (#25153 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21881+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/25190+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/25203+ 0.0 0.0
QA failures on gprd-cny (#25209 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/25211+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/25212+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/25213+ 0.0 0.0
2026-04-26: Elastic disk space utilization in l... (gitlab-com/gl-infra/production#21891 - closed) 0.0 0.0
Total 43.0 43.0

Additional incidents

Below is a list of production incidents created last week.

Click to expand
Issue
2026-04-26: Blackbox probe availability below 6... (gitlab-com/gl-infra/production#21892 - closed)
2026-04-25: Disk space on gitaly main stage pre... (gitlab-com/gl-infra/production#21890 - closed)
2026-04-25: gitaly unavailable - unable to hand... (gitlab-com/gl-infra/production#21889 - closed)
2026-04-25: High transaction rollback rate on p... (gitlab-com/gl-infra/production#21888 - closed)
2026-04-24: 5xx error rate on git service loadb... (gitlab-com/gl-infra/production#21886 - closed)
2026-04-24: Sidekiq queueing duration SLO viola... (gitlab-com/gl-infra/production#21883 - closed)
2026-04-24: Nonprod Elastic cluster unhealthy. (gitlab-com/gl-infra/production#21880 - closed)
2026-04-23: Increase in Errors for Anthropic Cl... (gitlab-com/gl-infra/production#21874 - closed)
2026-04-23: Web requests through workhorse have... (gitlab-com/gl-infra/production#21871 - closed)
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21870+
2026-04-23: Apdex SLO violation for code comple... (gitlab-com/gl-infra/production#21866)
2026-04-23: Sidekiq jobs on urgent-authorized-p... (gitlab-com/gl-infra/production#21863 - closed)
2026-04-23: High transaction rollback rate on p... (gitlab-com/gl-infra/production#21862 - closed)
2026-04-22: Workhorse web requests in us-east1-... (gitlab-com/gl-infra/production#21855 - closed)
2026-04-22: Workhorse API authentication reques... (gitlab-com/gl-infra/production#21854)
2026-04-22: 2026-04-22-data-server-rebuild-ansi... (gitlab-com/gl-infra/production#21852 - closed)
2026-04-22: Master Broken for Database Related ... (gitlab-com/gl-infra/production#21849 - closed)
2026-04-21: Disk space on gitaly node predicted... (gitlab-com/gl-infra/production#21848 - closed)
2026-04-21: Prometheus failing to scrape export... (gitlab-com/gl-infra/production#21843 - closed)
2026-04-21: Runners saas-linux-small-amd64 queu... (gitlab-com/gl-infra/production#21842 - closed)
2026-04-20: 2026-04-20: Data-Server Rebuild Ans... (gitlab-com/gl-infra/production#21837 - closed)
2026-04-20: At least 10% of chef-runs are faili... (gitlab-com/gl-infra/production#21833 - closed)
2026-04-20: Cloudflare issues currently impacti... (gitlab-com/gl-infra/production#21830 - closed)

Instructions

  • Review the Additional incidents list and add the Deploys-blocked and RootCause labels if required.
  • Retrigger this pipeline job to update this report and metrics with the updated values. Consider this will discard any manual changes added.
  • Update the "weekly overview" table of this issue to also include:
    • any blocking CRs
  • Update the Deployments metric review epic.
    • Add a new row to the Overview section: Copy and paste the information from the Overview section in this issue and link to this issue in the Breakdown of blockers column.
    • Update the Graph: Update the data on the spreadsheet and then update the graph on the Deployments metric review epic.

📣 The deployment blockers per week can now be visualized in the Deployment Blockers Dashboard.

Edited by Mayra Cabrera