Deployment Blockers - Week: 2026-06-22-2026-06-28

Overview

Start date End date Production deployments blocked for # of blockers
2026-06-22 2026-06-28 37.0 53

Blocker types overview

Blocker Type Time blocked on gstg Time blocked on gprd
RootCauseFlaky-Test 6.0 6.0
RootCauseConfig-Change 5.0 5.0
RootCauseDB-Migration 12.0 12.0
Change-request 14.0 14.0
Total 37.0 37.0

Weekly overview

Issue Blocker type gstg gprd
gitlab-com/gl-infra/production#22298 (closed) Change-request 14.0 14.0
2026-06-25: qa-smoke of post-deploy migration p... (gitlab-com/gl-infra/production#22379 - closed) RootCauseDB-Migration 12.0 12.0
2026-06-23: Omnibus build fails due to license ... (gitlab-com/gl-infra/production#22360 - closed) RootCauseConfig-Change 3.0 3.0
2026-06-23: git pushes are failing via ssh (gitlab-com/gl-infra/production#22364 - closed) RootCauseConfig-Change 2.0 2.0
QA failure: qa/specs/features/ee/browser_ui/2_p... (#27319 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: (#27318 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/9_tena... (#27204 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/9_tena... (#27209 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/ee/browser_ui/3_c... (#27212 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: (#27240 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#27235 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/ee/api/2_plan/epi... (#27288 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gprd-cny (#27234 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/ee/api/2_plan/epi... (#27243 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/api/10_software_s... (#27250 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: (#27251 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/27254+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27259+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27253+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27200+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27261+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27262+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27264+ 0.0 0.0
QA failure: qa/specs/features/browser_ui/2_plan... (#27267 - closed) 0.0 0.0
QA failure: qa/specs/features/ee/api/2_plan/epi... (#27272 - closed) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27274+ 0.0 0.0
QA failure: qa/specs/features/browser_ui/4_veri... (#27275 - closed) 0.0 0.0
QA failure: qa/specs/features/browser_ui/3_crea... (#27279 - closed) 0.0 0.0
QA failure: qa/specs/features/browser_ui/3_crea... (#27282 - closed) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27307+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27308+ 0.0 0.0
2026-06-28: Ongoing and user-visible, unauthent... (gitlab-com/gl-infra/production#22397 - closed) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27201+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27203+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27214+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27215+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27216+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27217+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27218+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27219+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27220+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27221+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27222+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27223+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27224+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27225+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27226+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27227+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27228+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27229+ 0.0 0.0
[C2] [GPRD] [2026-06-27 to 2026-06-30] - Upgrad... (gitlab-com/gl-infra/production#22363 - closed) Change-request 0.0 0.0
QA failure: qa/specs/features/browser_ui/10_sof... (#27230 - closed) 0.0 0.0
2026-06-23: Error pushing and pulling from GIt ... (gitlab-com/gl-infra/production#22365) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27242+ 0.0 0.0
Total 23.0 23.0

Additional incidents

Below is a list of production incidents created last week.

Click to expand
Issue
2026-06-28: At least 10% of chef-runs failing i... (gitlab-com/gl-infra/production#22399 - closed)
2026-06-28: runway_ingress apdex at 69.65% on d... (gitlab-com/gl-infra/production#22396 - closed)
2026-06-27: The Disk inode Utilization per Devi... (gitlab-com/gl-infra/production#22395 - closed)
2026-06-27: More than one postgres instance in ... (gitlab-com/gl-infra/production#22394 - closed)
2026-06-26: Restrict group access by IP address... (gitlab-com/gl-infra/production#22391 - closed)
2026-06-26: pipeline_processing_sidekiq_executi... (gitlab-com/gl-infra/production#22390 - closed)
2026-06-26: pipeline_processing_sidekiq_executi... (gitlab-com/gl-infra/production#22389 - closed)
2026-06-26: server_foundational error rate at 1... (gitlab-com/gl-infra/production#22388 - closed)
2026-06-26: Chef client failures at 10% in gprd (gitlab-com/gl-infra/production#22387 - closed)
2026-06-25: server_foundational error rate at 7... (gitlab-com/gl-infra/production#22384 - closed)
2026-06-25: Redis memory utilization nearing ma... (gitlab-com/gl-infra/production#22383 - closed)
2026-06-25: server_foundational error rate at 6... (gitlab-com/gl-infra/production#22381 - closed)
2026-06-25: The runner_api_requests SLI of the ... (gitlab-com/gl-infra/production#22380 - closed)
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22376+
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22368+
2026-06-23: QA fails on Release Environment 19.0 (gitlab-com/gl-infra/production#22362 - closed)
2026-06-23: server_foundational error rate at 6... (gitlab-com/gl-infra/production#22356 - closed)
2026-06-23: PostgreSQL LockManager LWLock conte... (gitlab-com/gl-infra/production#22355 - closed)
2026-06-22: Sidekiq queueing apdex at 67.82% on... (gitlab-com/gl-infra/production#22352 - closed)
2026-06-22: Blackbox probe availability for htt... (gitlab-com/gl-infra/production#22351 - closed)
2026-06-22: (OPS) The polling SLI of the ci-run... (gitlab-com/gl-infra/production#22349 - closed)
2026-06-22: gitlab-restore/postgres-gprd snitch... (gitlab-com/gl-infra/production#22348 - closed)

Instructions

  • Review the Additional incidents list and add the Deploys-blocked and RootCause labels if required.
  • Retrigger this pipeline job to update this report and metrics with the updated values. Consider this will discard any manual changes added.
  • Update the "weekly overview" table of this issue to also include:
    • any blocking CRs
  • Update the Deployments metric review epic.
    • Add a new row to the Overview section: Copy and paste the information from the Overview section in this issue and link to this issue in the Breakdown of blockers column.
    • Update the Graph: Update the data on the spreadsheet and then update the graph on the Deployments metric review epic.

📣 The deployment blockers per week can now be visualized in the Deployment Blockers Dashboard.

Edited by Mawreen Dela Cruz