Deployment Blockers - Week: 2026-10-05-2026-10-11

Overview

Start date End date Production deployments blocked for # of blockers
2026-10-05 2026-10-11 64.0 56

Blocker types overview

Blocker Type Time blocked on gstg Time blocked on gprd
Change-request 18.0 18.0
RootCauseFlaky-Test 13.0 13.0
RootCauseSoftware-Change 7.0 7.0
RootCauseFeature-Flag 7.0 7.0
Total 45.0 45.0

Weekly overview

Issue Blocker type gstg gprd
2026-10-06: Soft Production Change Lock for Git... (gitlab-com/gl-infra/production#23105 - closed) Change-request 17.0 17.0
2026-10-07: QA smoke tests on staging canary ar... (gitlab-com/gl-infra/production#23123 - closed) RootCauseFeature-Flag 7.0 7.0
2026-10-07: GitLab Rails Security Mirror Master... (gitlab-com/gl-infra/production#23114 - closed) RootCauseSoftware-Change 7.0 7.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23150+ 4.0 4.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23146+ 3.0 3.0
2026-10-08: CI job finalization & webhook delay... (gitlab-com/gl-infra/production#23131 - closed) 3.0 3.0
2026-10-09: GitLab.com hosted runner saas-linux... (gitlab-com/gl-infra/production#23149 - closed) 2.0 2.0
2026-10-09: Getting 503 across .com (gitlab-com/gl-infra/production#23141 - closed) 2.0 2.0
2026-10-09: broken dev.gitlab.org/gitlab/gitlab... (gitlab-com/gl-infra/production#23140 - closed) 2.0 2.0
2026-10-06: gprd deploy blocked (gitlab-com/gl-infra/production#23111 - closed) 2.0 2.0
[gprd] Move concurrency-limit deferred-job stor... (gitlab-com/gl-infra/production#23004 - closed) Change-request 1.0 1.0
2026-10-05: canonical -> security broken mirror... (gitlab-com/gl-infra/production#23095 - closed) 1.0 1.0
QA failure: qa/specs/features/browser_ui/10_sof... (#28019 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#28022 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/9_tena... (#28025 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#28026 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/9_tena... (#28028 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/9_tena... (#28029 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/9_tena... (#28030 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/ee/api/2_plan/epi... (#28031 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/api/10_software_s... (#28032 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/2_plan... (#27995 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#28035 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#28036 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/4_veri... (#28037 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#28038 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#28001 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#28002 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#28003 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#28004 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#28008 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#28009 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#28011 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/9_tena... (#28012 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#28014 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/9_tena... (#28015 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/10_sof... (#28018 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#28017 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/27998+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27999+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28000+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28020+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28021+ 0.0 0.0
2026-10-08: Delay job completion (gitlab-com/gl-infra/production#23133) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28005+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28006+ 0.0 0.0
QA failures on gstg-cny (#28007) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28016+ 0.0 0.0
QA failure: qa/specs/features/browser_ui/6_depl... (#28034) 0.0 0.0
QA failures on gstg-cny (#28033) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28023+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28010+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28024+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28013+ 0.0 0.0
2026-10-06: Soft PCL for GitLab Transcend India (gitlab-com/gl-infra/production#23101 - closed) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28027+ 0.0 0.0
Total 64.0 64.0

Additional incidents

Below is a list of production incidents created last week.

Click to expand
Issue
2026-10-11: Workhorse apdex SLO violation on we... (gitlab-com/gl-infra/production#23151 - closed)
2026-10-09: Depdency proxy errors increasing (gitlab-com/gl-infra/production#23148 - closed)
2026-10-09: grafana_datasources error rate at 1... (gitlab-com/gl-infra/production#23144 - closed)
2026-10-08: customers_dot_requests apdex at 89.... (gitlab-com/gl-infra/production#23142 - closed)
2026-10-08: Loadbalancer error rate at 0.06944%... (gitlab-com/gl-infra/production#23139 - closed)
2026-10-08: 2026-10-08: Data Server Rebuild rep... (gitlab-com/gl-infra/production#23137 - closed)
2026-10-08: gprd-deploy-failed-with-bad-helm-state (gitlab-com/gl-infra/production#23136 - closed)
2026-10-08: The ci_runner_jobs SLI of the ci-ru... (gitlab-com/gl-infra/production#23135)
2026-10-08: PubSub queuing high on pubsub-rails... (gitlab-com/gl-infra/production#23134 - closed)
2026-10-08: Sidekiq queueing apdex 99.32% on ur... (gitlab-com/gl-infra/production#23132)
2026-10-08: amd64 saas linux small - nearing ca... (gitlab-com/gl-infra/production#23129 - closed)
2026-10-07: customers_dot_requests apdex at 91.... (gitlab-com/gl-infra/production#23128)
2026-10-07: Pods stuck Unschedulable in ops-cen... (gitlab-com/gl-infra/production#23126 - closed)
2026-10-07: artifacts_cdn error rate at 0.576% ... (gitlab-com/gl-infra/production#23121 - closed)
2026-10-07: rate-limit-brownout-ff-problems (gitlab-com/gl-infra/production#23120 - closed)
2026-10-07: PubSub queuing high on pubsub-workh... (gitlab-com/gl-infra/production#23119 - closed)
2026-10-07: Customers reporting extended job qu... (gitlab-com/gl-infra/production#23117 - closed)
2026-10-06: The ci_runner_jobs SLI of the ci-ru... (gitlab-com/gl-infra/production#23108 - closed)
2026-10-06: Chef client failures at 10% in gprd (gitlab-com/gl-infra/production#23106 - closed)
2026-10-06: Sentry (new-sentry.gitlab.net) not ... (gitlab-com/gl-infra/production#23104 - closed)
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23100+
2026-09-28: PubSub queuing high on pubsub-workh... (gitlab-com/gl-infra/production#23098 - closed)
2026-10-05: The ci_runner_jobs SLI of the ci-ru... (gitlab-com/gl-infra/production#23097 - closed)
2026-10-05: pulp_nginx apdex at 98.46% (gitlab-com/gl-infra/production#23093 - closed)

Instructions

  • Review the Additional incidents list and add the Deploys-blocked and RootCause labels if required.
  • Retrigger this pipeline job to update this report and metrics with the updated values. Consider this will discard any manual changes added.
  • Update the "weekly overview" table of this issue to also include:
    • any blocking CRs
  • Update the Deployments metric review epic.
    • Add a new row to the Overview section: Copy and paste the information from the Overview section in this issue and link to this issue in the Breakdown of blockers column.
    • Update the Graph: Update the data on the spreadsheet and then update the graph on the Deployments metric review epic.

📣 The deployment blockers per week can now be visualized in the Deployment Blockers Dashboard.

Edited by Manjunath Bhadrannavar