Deployment Blockers - Week: 2026-08-24-2026-08-30

Overview

Start date End date Production deployments blocked for # of blockers
2026-08-24 2026-08-30 38.5 52

Blocker types overview

Blocker Type Time blocked on gstg Time blocked on gprd
RootCauseFlaky-Test 9.5 9.5
Change-request 2.0 2.0
RootCauseExternal-Dependency 4.0 4.0
RootCauseSoftware-Change 10.0 10.0
RootCauseConfig-Change 7.0 8.0
RootCauseFeature-Flag 3.0 3.0
RootCauseIndeterminate 2.0 2.0
Total 37.5 38.5

Weekly overview

Issue Blocker type gstg gprd
2026-08-25: 500s in ops.gitlab.net (gitlab-com/gl-infra/production#22785 - closed) RootCauseConfig-Change 7.0 8.0
2026-08-24: KAS deployments failing (gitlab-com/gl-infra/production#22777 - closed) RootCauseSoftware-Change 8.0 8.0
2026-08-24: Pods slow to become ready due to us... (gitlab-com/gl-infra/production#22773 - closed) RootCauseExternal-Dependency 4.0 4.0
2026-08-25: QA smoke jobs can't get started (gitlab-com/gl-infra/production#22789 - closed) RootCauseFeature-Flag 3.0 3.0
2026-08-26: SSCS smoke tests failing on gprd-cn... (gitlab-com/gl-infra/production#22799 - closed) RootCauseIndeterminate 2.0 2.0
2026-08-28: Migrate ops.gitlab.net DB (ops-cent... (gitlab-com/gl-infra/production#22771 - closed) Change-request 2.0 2.0
2026-08-26: `milestone/assign_milestone_spec.rb... (gitlab-com/gl-infra/production#22800 - closed) RootCauseSoftware-Change 2.0 2.0
QA failure: qa/specs/features/browser_ui/10_sof... (#27781 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#27756 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/2_plan... (#27757 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gprd-cny (#27758 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/2_plan... (#27764 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/2_plan... (#27765 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#27766 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/ee/browser_ui/18_... (#27767 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/ee/api/10_softwar... (#27768 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gprd-cny (#27771 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/api/10_software_s... (#27772 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: (#27775 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#27776 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/ee/api/2_plan/epi... (#27780 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: (#27738 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#27743 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#27746 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/2_plan... (#27747 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gprd-cny (#27752 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22783+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27741+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27779+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27778+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27777+ 0.0 0.0
Pods slow to become ready due to us-east1-c GCE... (#27742 - closed) 0.0 0.0
QA failures on gstg-cny (#27755 - closed) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27774+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27773+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27744+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27745+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27770+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27769+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27754+ 0.0 0.0
2026-08-27: patroni-sec CPU saturated (gitlab-com/gl-infra/production#22807 - closed) 0.0 0.0
QA failures on gstg-cny (#27753 - closed) 0.0 0.0
QA failure: qa/specs/features/browser_ui/10_sof... (#27748 - closed) 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22802+ 0.0 0.0
QA failure: qa/specs/features/browser_ui/10_sof... (#27749 - closed) 0.0 0.0
QA failures on gprd-cny (#27750 - closed) 0.0 0.0
QA failure: qa/specs/features/browser_ui/10_sof... (#27751 - closed) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27763+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27762+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27759+ 0.0 0.0
QA failures on gprd-cny (#27761 - closed) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27760+ 0.0 0.0
Total 37.5 38.5

Additional incidents

Below is a list of production incidents created last week.

Click to expand
Issue
2026-08-29: Loadbalancer error rate at 0.1554% ... (gitlab-com/gl-infra/production#22826 - closed)
2026-08-29: runway_ingress apdex 69.54% on duo-... (gitlab-com/gl-infra/production#22825 - closed)
2026-08-29: KAS job_router error rate violating... (gitlab-com/gl-infra/production#22824 - closed)
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22820+
2026-08-27: Instances per VPC Network reached i... (gitlab-com/gl-infra/production#22819 - closed)
2026-08-27: Upstream ubuntu mirror errors in GS... (gitlab-com/gl-infra/production#22818 - closed)
2026-08-27: Runaway Dead Tuple Growth across Se... (gitlab-com/gl-infra/production#22817 - closed)
2026-08-27: workhorse apdex SLO violation in us... (gitlab-com/gl-infra/production#22816)
2026-08-27: The ci_runner_jobs SLI of the ci-ru... (gitlab-com/gl-infra/production#22815 - closed)
2026-08-27: GCP Quota utilization near capacity... (gitlab-com/gl-infra/production#22814 - closed)
2026-08-27: Alertmanager slack integration prob... (gitlab-com/gl-infra/production#22813 - closed)
2026-08-27: gitaly goserver traffic cessation o... (gitlab-com/gl-infra/production#22811 - closed)
2026-08-27: GCP quota utilization on SSD_TOTAL_... (gitlab-com/gl-infra/production#22810 - closed)
2026-08-27: Web service loadbalancer error rate... (gitlab-com/gl-infra/production#22806 - closed)
2026-08-27: web loadbalancer error rate violati... (gitlab-com/gl-infra/production#22803)
2026-08-27: Postgres int4 ID capacity saturatio... (gitlab-com/gl-infra/production#22801 - closed)
2026-08-26: loadbalancer SLI error rate at 0.06... (gitlab-com/gl-infra/production#22798)
2026-08-26: sidekiq_queueing apdex 96.98% on lo... (gitlab-com/gl-infra/production#22797 - closed)
2026-08-26: Users reporting 404s viewing shared... (gitlab-com/gl-infra/production#22795 - closed)
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22794+
2026-08-25: Siphon logical replication lag on p... (gitlab-com/gl-infra/production#22791 - closed)
2026-08-25: check-packages-functionality jobs a... (gitlab-com/gl-infra/production#22790 - closed)
2026-08-25: Chef client hasn't converged succes... (gitlab-com/gl-infra/production#22788 - closed)
2026-08-25: The ci_runner_jobs SLI of the ci-ru... (gitlab-com/gl-infra/production#22786 - closed)
2026-08-25: PGBouncer Client Connections per Pr... (gitlab-com/gl-infra/production#22782)
2026-08-25: rails_primary_sql apdex at 85.17% o... (gitlab-com/gl-infra/production#22781)
2026-08-25: Sidekiq execution apdex at 67.55% o... (gitlab-com/gl-infra/production#22780)
2026-08-24: Workhorse apdex violating SLO in cn... (gitlab-com/gl-infra/production#22778)
2026-08-24: pulp_nginx apdex at 95.89% on main ... (gitlab-com/gl-infra/production#22774 - closed)
2026-08-24: sidekiq_queueing apdex at 23.87% on... (gitlab-com/gl-infra/production#22772 - closed)
2026-08-24: gitlab-restore/postgres-gprd missing (gitlab-com/gl-infra/production#22770 - closed)

Instructions

  • Review the Additional incidents list and add the Deploys-blocked and RootCause labels if required.
  • Retrigger this pipeline job to update this report and metrics with the updated values. Consider this will discard any manual changes added.
  • Update the "weekly overview" table of this issue to also include:
    • any blocking CRs
  • Update the Deployments metric review epic.
    • Add a new row to the Overview section: Copy and paste the information from the Overview section in this issue and link to this issue in the Breakdown of blockers column.
    • Update the Graph: Update the data on the spreadsheet and then update the graph on the Deployments metric review epic.

📣 The deployment blockers per week can now be visualized in the Deployment Blockers Dashboard.

Edited by Mawreen Dela Cruz