Deployment Blockers - Week: 2026-09-07-2026-09-13

Overview

Start date End date Production deployments blocked for # of blockers
2026-09-07 2026-09-13 49.0 45

Blocker types overview

Blocker Type Time blocked on gstg Time blocked on gprd
RootCauseDB-Migration 2.0 2.0
RootCauseFlaky-Test 8.0 8.0
RootCauseMalicious-Traffic 0.5 0.5
RootCauseSoftware-Change 29.5 29.5
RootCauseConfig-Change 3.0 3.0
RootCauseSaturation 6.0 6.0
Total 49.0 49.0

Weekly overview

Issue Blocker type gstg gprd
2026-09-10: Deployment blocker issue for master... (#27845 - closed) RootCauseSoftware-Change 18.0 18.0
2026-09-10: k8s deployment failing on gstg-cny (gitlab-com/gl-infra/production#22909 - closed) RootCauseSoftware-Change 6.0 6.0
2026-09-11: Omnibus packages are failing for au... (gitlab-com/gl-infra/production#22920 - closed) RootCauseSoftware-Change 5.0 5.0
2026-09-08: k8s deployments are stuck in gprd-cny (gitlab-com/gl-infra/production#22884 - closed) RootCauseConfig-Change 3.0 3.0
2026-09-09: Redis memory utilization near max o... (gitlab-com/gl-infra/production#22897 - closed) RootCauseSaturation 2.0 2.0
2026-09-10: Sidekiq jobs slow to process and be... (gitlab-com/gl-infra/production#22904 - closed) RootCauseSaturation 2.0 2.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22914+ RootCauseSaturation 2.0 2.0
2026-09-07: gprd post-deploy migration failed t... (gitlab-com/gl-infra/production#22873 - closed) RootCauseDB-Migration 2.0 2.0
QA failure: (#27853 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: (#27852 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/10_sof... (#27851 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/10_sof... (#27850 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/ee/api/2_plan/epi... (#27849 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/api/3_create/merg... (#27848 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/10_sof... (#27841 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: (#27854 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#27840 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#27815 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#27816 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/2_plan... (#27819 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#27820 - closed) RootCauseFlaky-Test 0.5 0.5
2026-09-08: Some catalog agents are not working (gitlab-com/gl-infra/production#22880 - closed) RootCauseSoftware-Change 0.5 0.5
2026-09-07: 68.9% of chat workflows are stopped (gitlab-com/gl-infra/production#22875 - closed) RootCauseMalicious-Traffic 0.5 0.5
QA failure: qa/specs/features/ee/api/2_plan/epi... (#27822 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/api/3_create/merg... (#27823 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#27824 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/27857+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27856+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27855+ 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22876+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27825+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27826+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27847+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27846+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27827+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27844+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27843+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27842+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27828+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27834+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27839+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27838+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27835+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27837+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27836+ 0.0 0.0
Total 49.0 49.0

Additional incidents

Below is a list of production incidents created last week.

Click to expand
Issue
2026-09-13: server_chat error rate 6.043% above... (gitlab-com/gl-infra/production#22926 - closed)
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22925+
2026-09-12: server_chat error rate at 6.337% on... (gitlab-com/gl-infra/production#22924 - closed)
2026-09-11: Siphon logical replication lag on p... (gitlab-com/gl-infra/production#22922 - closed)
2026-09-11: Workhorse apdex SLO violation in cn... (gitlab-com/gl-infra/production#22918 - closed)
2026-09-11: Smoke tests are failing on the pre ... (gitlab-com/gl-infra/production#22917 - closed)
2026-09-11: pipeline_processing_sidekiq_executi... (gitlab-com/gl-infra/production#22916 - closed)
2026-09-09: The ci_runner_jobs SLI of the ci-ru... (gitlab-com/gl-infra/production#22912 - closed)
2026-09-10: web_pages_server error rate at 0.73... (gitlab-com/gl-infra/production#22911 - closed)
2026-09-10: Comments are not loading on MRs in ... (gitlab-com/gl-infra/production#22908)
2026-09-10: Last Patroni GCS snapshot did not r... (gitlab-com/gl-infra/production#22906 - closed)
2026-09-10: Persistent shared_runner_job_queue_... (gitlab-com/gl-infra/production#22905)
2026-09-10: gstg-cny canary readiness check fai... (gitlab-com/gl-infra/production#22902 - closed)
2026-09-09: Duo Workflow Service unable to perf... (gitlab-com/gl-infra/production#22900 - closed)
2026-09-09: http-router worker_requests traffic... (gitlab-com/gl-infra/production#22895 - closed)
2026-09-09: Blackbox probe availability for cus... (gitlab-com/gl-infra/production#22892 - closed)
2026-09-09: Loadbalancer error rate at 0.06053%... (gitlab-com/gl-infra/production#22891 - closed)
2026-09-09: DAP flows fail in gitlab-org/gitlab... (gitlab-com/gl-infra/production#22890 - closed)
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22886+
2026-09-08: The ci_runner_jobs SLI of the ci-ru... (gitlab-com/gl-infra/production#22883 - closed)
2026-09-07: runway_ingress apdex at 67.33% (gitlab-com/gl-infra/production#22881 - closed)
2026-09-08: Postgres int4 ID capacity saturatio... (gitlab-com/gl-infra/production#22879 - closed)
2026-09-07: Disk space utilization for postgres... (gitlab-com/gl-infra/production#22878 - closed)
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22877+
2026-09-07: web loadbalancer error rate SLO vio... (gitlab-com/gl-infra/production#22874 - closed)

Instructions

  • Review the Additional incidents list and add the Deploys-blocked and RootCause labels if required.
  • Retrigger this pipeline job to update this report and metrics with the updated values. Consider this will discard any manual changes added.
  • Update the "weekly overview" table of this issue to also include:
    • any blocking CRs
  • Update the Deployments metric review epic.
    • Add a new row to the Overview section: Copy and paste the information from the Overview section in this issue and link to this issue in the Breakdown of blockers column.
    • Update the Graph: Update the data on the spreadsheet and then update the graph on the Deployments metric review epic.

📣 The deployment blockers per week can now be visualized in the Deployment Blockers Dashboard.

Edited by Mawreen Dela Cruz