Deployment Blockers - Week: 2026-08-17-2026-08-23

Overview

Start date End date Production deployments blocked for # of blockers
2026-08-17 2026-08-23 63.0 71

Blocker types overview

Blocker Type Time blocked on gstg Time blocked on gprd
Change-request 4.0 4.0
RootCauseFlaky-Test 16.0 16.0
RootCauseSoftware-Change 6.0 6.0
RootCauseSaturation 17.0 17.0
RootCauseDB-Migration 16.0 16.0
RootCauseIndeterminate 4.0 4.0
Total 63.0 63.0

Weekly overview

Issue Blocker type gstg gprd
2026-08-18: Staging qa-smoke failures block dep... (gitlab-com/gl-infra/production#22725 - closed) RootCauseDB-Migration 16.0 16.0
2026-08-17: Gitaly Resource Exhaustion & Increa... (gitlab-com/gl-infra/production#22722 - closed) RootCauseSaturation 8.0 8.0
2026-08-17: Project/Group deletions are failing... (gitlab-com/gl-infra/production#22714 - closed) RootCauseSoftware-Change 6.0 6.0
2026-08-20: Gitaly concurrency saturation (gitlab-com/gl-infra/production#22750 - closed) RootCauseSaturation 6.0 6.0
2026-08-18: Loadbalancer error rate violating SLO (gitlab-com/gl-infra/production#22729 - closed) RootCauseIndeterminate 4.0 4.0
2026-08-18: gprd prometheus agent stuck in rest... (gitlab-com/gl-infra/production#22730 - closed) RootCauseSaturation 2.0 2.0
2026-08-23: Migrate ops.gitlab.net (gitlab-ops)... (gitlab-com/gl-infra/production#22684 - closed) Change-request 2.0 2.0
2026-08-20: Gitaly IO saturation (gitlab-com/gl-infra/production#22754 - closed) RootCauseSaturation 1.0 1.0
Run PDM `RemoveTmpBigintFkForDeploymentsPhaseTw... (gitlab-com/gl-infra/production#22716 - closed) Change-request 1.0 1.0
FY27-Q3 HAProxy DR Gameday (outage in us-east1-b) (gitlab-com/gl-infra/production#22670 - closed) Change-request 1.0 1.0
QA failure: (#27674 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/10_sof... (#27696 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/ee/browser_ui/18_... (#27731 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/2_plan... (#27699 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#27700 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gprd-cny (#27701 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/api/10_software_s... (#27729 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#27718 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#27704 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/api/10_software_s... (#27727 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#27707 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/6_depl... (#27724 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/2_plan... (#27717 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gprd-cny (#27712 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/6_depl... (#27714 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/ee/browser_ui/3_c... (#27721 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/9_tena... (#27716 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/ee/browser_ui/2_p... (#27675 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gprd-cny (#27677 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gprd-cny (#27678 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: (#27679 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gprd-cny (#27680 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: (#27681 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/2_plan... (#27682 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/9_tena... (#27736 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#27685 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: (#27694 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/6_depl... (#27692 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#27691 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/2_plan... (#27689 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#27687 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/9_tena... (#27735 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/27725+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27737+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27723+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27734+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27722+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27726+ 0.0 0.0
2026-08-21: Edge rule blocking HTTPS clones (gitlab-com/gl-infra/production#22762 - closed) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27732+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27720+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27728+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27733+ 0.0 0.0
QA failures on gstg-cny (#27683 - closed) 0.0 0.0
QA failures on gstg-cny (#27684 - closed) 0.0 0.0
QA failures on gstg-cny (#27686 - closed) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27688+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27690+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27695+ 0.0 0.0
QA failure: qa/specs/features/api/10_software_s... (#27697 - closed) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27698+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27702+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27703+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27705+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27708+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27709+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27710+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27711+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27713+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27715+ 0.0 0.0
2026-08-20: pg_int4_id saturation exceeding SLO... (gitlab-com/gl-infra/production#22753 - closed) RootCauseSaturation 0.0 0.0
Total 63.0 63.0

Additional incidents

Below is a list of production incidents created last week.

Click to expand
Issue
2026-08-21: workhorse apdex violating SLO in cn... (gitlab-com/gl-infra/production#22767)
2026-08-21: Blackbox probes failing for handboo... (gitlab-com/gl-infra/production#22766 - closed)
2026-08-21: The loadbalancer SLI of the ci-runn... (gitlab-com/gl-infra/production#22765)
2026-08-21: web loadbalancer and workhorse erro... (gitlab-com/gl-infra/production#22763 - closed)
2026-08-20: The saas_linux_runner_image_pull_fa... (gitlab-com/gl-infra/production#22751 - closed)
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22748+
2026-08-19: Mimir has gaps when it's busy (gitlab-com/gl-infra/production#22747 - closed)
2026-08-19: Container Registry GCS blob corrupt... (gitlab-com/gl-infra/production#22746 - closed)
2026-08-19: pgbouncer sidekiq low-urgency satur... (gitlab-com/gl-infra/production#22741 - closed)
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22740+
2026-08-19: Blackbox probe availability for cus... (gitlab-com/gl-infra/production#22737 - closed)
2026-08-19: 19.3 stable branch failure after De... (gitlab-com/gl-infra/production#22736 - closed)
2026-08-19: static-analysis job is failing in 1... (gitlab-com/gl-infra/production#22735 - closed)
2026-08-18: The Disk inode Utilization per Devi... (gitlab-com/gl-infra/production#22733 - closed)
2026-08-18: The Disk Space Utilization per Devi... (gitlab-com/gl-infra/production#22732)
2026-08-18: The ci_runner_jobs SLI of the ci-ru... (gitlab-com/gl-infra/production#22731)
2026-08-18: staging.gitlab.com is down (gitlab-com/gl-infra/production#22728)
2026-08-17: Kube Persistent Volume Claim inode ... (gitlab-com/gl-infra/production#22724 - closed)
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22723+
2026-08-17: External DNS stale on gprd-gitlab-gke (gitlab-com/gl-infra/production#22718 - closed)
2026-08-17: Kube Persistent Volume Claim inode ... (gitlab-com/gl-infra/production#22715 - closed)

Instructions

  • Review the Additional incidents list and add the Deploys-blocked and RootCause labels if required.
  • Retrigger this pipeline job to update this report and metrics with the updated values. Consider this will discard any manual changes added.
  • Update the "weekly overview" table of this issue to also include:
    • any blocking CRs
  • Update the Deployments metric review epic.
    • Add a new row to the Overview section: Copy and paste the information from the Overview section in this issue and link to this issue in the Breakdown of blockers column.
    • Update the Graph: Update the data on the spreadsheet and then update the graph on the Deployments metric review epic.

📣 The deployment blockers per week can now be visualized in the Deployment Blockers Dashboard.

Edited by Maina Ng'ang'a