Deployment Blockers - Week: 2025-09-22-2025-09-28

Overview

Start date End date Production deployments blocked for # of blockers
2025-09-22 2025-09-28 47.5 22

Blocker types overview

Blocker Type Time blocked on gstg Time blocked on gprd
RootCauseConfig-Change 24.0 24.0
RootCauseFlaky-Test 3.5 3.5
RootCauseSoftware-Change 10.0 10.0
RootCauseExternal-Dependency 4.0 4.0
Total 41.5 41.5

Weekly overview

Issue Blocker type gstg gprd
2025-09-22: Deployments to gitlab.com missing c... (gitlab-com/gl-infra/production#20585 - closed) RootCauseConfig-Change 24.0 24.0
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20592+ RootCauseSoftware-Change 10.0 10.0
2025-09-23: Merge Request home page is empty (gitlab-com/gl-infra/production#20599 - closed) 1.0 1.0
2025-09-24: 500 Errors in gstg Observed in Smok... (gitlab-com/gl-infra/production#20602 - closed) 5.0 5.0
2025-09-24: Image pull failures on SaaS Linux r... (gitlab-com/gl-infra/production#20607 - closed) RootCauseExternal-Dependency 4.0 4.0
QA failure: qa/specs/features/browser_ui/2_plan... (#21413 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/2_plan... (#21416 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/ee/api/10_softwar... (#21432 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#21434 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/2_plan... (#21444 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/2_plan... (#21456 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#21472 - closed) RootCauseFlaky-Test 0.5 0.5
2025-09-22: Epic Board doesn't render on gitlab... (gitlab-com/gl-infra/production#20588 - closed) 0.0 0.0
2025-09-23: GitLab Runner API 429 (gitlab-com/gl-infra/production#20593 - closed) 0.0 0.0
QA failures on gprd-cny (#21419 - closed) 0.0 0.0
QA failure: qa/specs/features/browser_ui/10_sof... (#21420 - closed) 0.0 0.0
QA failures on gstg-cny (#21450 - closed) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/21454+ 0.0 0.0
QA failures on gstg-cny (#21458 - closed) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/21463+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/21466+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/21490+ 0.0 0.0
Total 47.5 47.5

Additional incidents

Below is a list of production incidents created last week.

Click to expand
Issue
2025-09-25: Sidekiq queueing SLO violation on l... (gitlab-com/gl-infra/production#20615 - closed)
2025-09-25: Sidekiq queueing SLO violation on l... (gitlab-com/gl-infra/production#20614 - closed)
2025-09-25: CI/CD variable environment scope is... (gitlab-com/gl-infra/production#20613 - closed)
2025-09-24: Apdex SLO violation for SQL transac... (gitlab-com/gl-infra/production#20605 - closed)
2025-09-24: Patroni GCS snapshot did not run on... (gitlab-com/gl-infra/production#20604 - closed)
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20600+
2025-09-23: PGBouncer client connections per pr... (gitlab-com/gl-infra/production#20596 - closed)
2025-09-23: Workspace Reconciliation Failures D... (gitlab-com/gl-infra/production#20594 - closed)
2025-09-22: Wiki pages not rendering on Canary (gitlab-com/gl-infra/production#20591 - closed)
2025-09-22: gprd k8s helm deploy failing (gitlab-com/gl-infra/production#20589 - closed)
2025-09-22: High error rate for job polling on ... (gitlab-com/gl-infra/production#20586 - closed)

Instructions

  • Review the Additional incidents list and add the Deploys-blocked and RootCause labels if required.
  • Retrigger this pipeline job to update this report and metrics with the updated values. Consider this will discard any manual changes added.
  • Update the "weekly overview" table of this issue to also include:
    • any blocking CRs
  • Update the Deployments metric review epic.
    • Add a new row to the Overview section: Copy and paste the information from the Overview section in this issue and link to this issue in the Breakdown of blockers column.
    • Update the Graph: Update the data on the spreadsheet and then update the graph on the Deployments metric review epic.

📣 The deployment blockers per week can now be visualized in the Deployment Blockers Dashboard.

Edited by Reuben Pereira