Deployment Blockers - Week: 2026-10-19-2026-10-25

Overview

Start date End date Production deployments blocked for # of blockers
2026-10-19 2026-10-25 39.0 72

Blocker types overview

Blocker Type Time blocked on gstg Time blocked on gprd
RootCauseFlaky-Test 16.0 16.0
RootCauseSoftware-Change 20.0 20.0
RootCauseSaturation 3.0 3.0
Total 39.0 39.0

Weekly overview

Issue Blocker type gstg gprd
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23292+ RootCauseSoftware-Change 9.0 9.0
2026-10-19: Database migration failing on produ... (gitlab-com/gl-infra/production#23284 - closed) RootCauseSoftware-Change 7.0 7.0
2026-10-19: ai-assisted rails_request error rat... (gitlab-com/gl-infra/production#23280 - closed) RootCauseSoftware-Change 4.0 4.0
2026-10-20: QA tests failing due to connection ... (gitlab-com/gl-infra/production#23294 - closed) RootCauseSaturation 3.0 3.0
QA failures on gstg-cny (#28113 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/ee/browser_ui/2_p... (#28154 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/api/10_software_s... (#28114 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gprd-cny (#28149 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#28116 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/api/10_software_s... (#28117 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/9_tena... (#28118 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/2_plan... (#28119 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: (#28120 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/4_veri... (#28121 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: (#28122 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#28123 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/4_veri... (#28124 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: (#28125 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#28134 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: (#28128 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: (#28132 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#28130 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#28108 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: (#28155 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#28135 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#28137 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/8_moni... (#28104 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/2_plan... (#28090 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/api/10_software_s... (#28147 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/api/3_create/merg... (#28145 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/api/10_software_s... (#28142 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: (#28099 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/9_tena... (#28098 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#28097 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: (#28096 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: (#28094 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/28089+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28146+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28148+ 0.0 0.0
2026-10-23: gitlab.com is either slow or not lo... (gitlab-com/gl-infra/production#23320 - closed) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28150+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28151+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28153+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28156+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28157+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28158+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28159+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28160+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28161+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28162+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28163+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28164+ 0.0 0.0
QA failure: qa/specs/features/ee/browser_ui/18_... (#28092 - closed) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28101+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28102+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28103+ 0.0 0.0
2026-10-20: 500 errors on GitLab SaaS when acce... (gitlab-com/gl-infra/production#23288 - closed) 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23289+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28105+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28106+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28107+ 0.0 0.0
2026-10-20: Customer Portal Returning 404 (gitlab-com/gl-infra/production#23297 - closed) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28115+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28126+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28129+ 0.0 0.0
2026-10-22: Gitaly not available (gitlab-com/gl-infra/production#23317) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28136+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28139+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28140+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28141+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28143+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28144+ 0.0 0.0
Total 39.0 39.0

Additional incidents

Below is a list of production incidents created last week.

Click to expand
Issue
2026-10-23: Pods unschedulable in gprd-us-east1... (gitlab-com/gl-infra/production#23327 - closed)
2026-10-25: narrow_salmon_gecko secondary insta... (gitlab-com/gl-infra/production#23326 - closed)
2026-10-24: web loadbalancer error rate SLO vio... (gitlab-com/gl-infra/production#23324 - closed)
2026-10-24: web loadbalancer error rate violati... (gitlab-com/gl-infra/production#23323 - closed)
2026-10-23: PostgreSQL BufferMapping LWLock con... (gitlab-com/gl-infra/production#23322 - closed)
2026-10-23: gprd deploy failed at all zones (gitlab-com/gl-infra/production#23319 - closed)
2026-10-22: helm stuck in upgrade state blockin... (gitlab-com/gl-infra/production#23318 - closed)
2026-10-22: PGBouncer Client Connections per Pr... (gitlab-com/gl-infra/production#23314)
2026-10-22: Data-Server Rebuild Ansible | Faile... (gitlab-com/gl-infra/production#23312 - closed)
2026-10-21: Error while pulling images from git... (gitlab-com/gl-infra/production#23311)
2026-10-21: DB connection pool saturation trigg... (gitlab-com/gl-infra/production#23310 - closed)
2026-10-21: Webook deliveries delayed (gitlab-com/gl-infra/production#23305)
2026-10-21: The ci_runner_jobs SLI of the ci-ru... (gitlab-com/gl-infra/production#23303 - closed)
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23302+
2026-10-20: The ci_runner_jobs SLI of the ci-ru... (gitlab-com/gl-infra/production#23295 - closed)
2026-10-20: kube Persistent Volume Claim inode ... (gitlab-com/gl-infra/production#23293 - closed)
2026-10-19: Puma services fail to reload on CDo... (gitlab-com/gl-infra/production#23282 - closed)

Instructions

  • Review the Additional incidents list and add the Deploys-blocked and RootCause labels if required.
  • Retrigger this pipeline job to update this report and metrics with the updated values. Consider this will discard any manual changes added.
  • Update the "weekly overview" table of this issue to also include:
    • any blocking CRs
  • Update the Deployments metric review epic.
    • Add a new row to the Overview section: Copy and paste the information from the Overview section in this issue and link to this issue in the Breakdown of blockers column.
    • Update the Graph: Update the data on the spreadsheet and then update the graph on the Deployments metric review epic.

📣 The deployment blockers per week can now be visualized in the Deployment Blockers Dashboard.

Edited by Dat Tang