Deployment Blockers - Week: 2026-09-28-2026-10-04

Overview

Start date End date Production deployments blocked for # of blockers
2026-09-28 2026-10-04 47.5 82

Blocker types overview

Blocker Type Time blocked on gstg Time blocked on gprd
RootCauseFlaky-Test 19.5 19.5
RootCauseExternal-Dependency 3.0 3.0
Total 22.5 22.5

Weekly overview

Issue Blocker type gstg gprd
2026-09-29: Unhealthy GitLab Deployment Health ... (gitlab-com/gl-infra/production#23051 - closed) 20.0 20.0
2026-10-01: Failing CNG jobs blocking auto-depl... (gitlab-com/gl-infra/production#23071 - closed) 3.0 3.0
2026-09-30: GitLab Email delivery to Yahoo and ... (gitlab-com/gl-infra/production#23061) RootCauseExternal-Dependency 3.0 3.0
2026-09-29: Broken auto-deploy (gstg-cny) Click... (gitlab-com/gl-infra/production#23053 - closed) 2.0 2.0
QA failures on gstg-cny (#27993 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/api/10_software_s... (#27977 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/10_sof... (#27976 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/api/10_software_s... (#27975 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#27974 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/ee/api/2_plan/epi... (#27973 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: (#27972 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/2_plan... (#27970 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: (#27945 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/9_tena... (#27946 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#27960 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/8_moni... (#27968 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#27967 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/2_plan... (#27950 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#27966 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/2_plan... (#27953 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/2_plan... (#27963 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/2_plan... (#27957 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/2_plan... (#27958 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#27992 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/2_plan... (#27962 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/10_sof... (#27920 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gprd-cny (#27989 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: (#27923 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#27988 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/2_plan... (#27987 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#27986 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/2_plan... (#27961 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/2_plan... (#27984 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#27983 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/10_sof... (#27982 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: (#27981 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/4_veri... (#27932 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#27980 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#27979 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#27935 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#27936 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#27978 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#27937 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23013+ Change-request 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27964+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27965+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27969+ 0.0 0.0
QA failure: qa/specs/features/api/10_software_s... (#27985 - closed) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27990+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27991+ 0.0 0.0
QA failures on gstg-cny (#27994 - closed) 0.0 0.0
2026-09-28: Canonical -> Security mirroring is ... (gitlab-com/gl-infra/production#23031 - closed) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27917+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27918+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27919+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27921+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27924+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27925+ 0.0 0.0
QA failure: qa/specs/features/browser_ui/9_tena... (#27926 - closed) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27927+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27928+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27929+ 0.0 0.0
QA failure: qa/specs/features/ee/browser_ui/2_p... (#27930 - closed) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27931+ 0.0 0.0
QA failure: qa/specs/features/browser_ui/10_sof... (#27933 - closed) 0.0 0.0
QA failures on gstg-cny (#27934 - closed) 0.0 0.0
2026-09-29: gstg-cny smoke test failures blocki... (gitlab-com/gl-infra/production#23044 - closed) 0.0 0.0
QA failure: qa/specs/features/browser_ui/10_sof... (#27938 - closed) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27939+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27940+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27941+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27942+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27943+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27944+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27947+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27948+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27949+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27951+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27952+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27956+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27959+ 0.0 0.0
2026-10-02: Test Rails 8.0 Rollout (gstg-cny, g... (gitlab-com/gl-infra/production#23072 - closed) Change-request 0.0 0.0
Total 47.5 47.5

Additional incidents

Below is a list of production incidents created last week.

Click to expand
Issue
2026-10-04: workhorse_auth_api error rate viola... (gitlab-com/gl-infra/production#23092 - closed)
2026-10-04: Dead Man's Snitch missing for gitla... (gitlab-com/gl-infra/production#23091 - closed)
2026-10-02: 500 errors on project for loud_beig... (gitlab-com/gl-infra/production#23088)
2026-10-02: Pods stuck Unschedulable in ops-cen... (gitlab-com/gl-infra/production#23087 - closed)
2026-10-02: 2026-10-02: Data-Server Rebuild Ans... (gitlab-com/gl-infra/production#23086 - closed)
2026-10-02: grafana_datasources error rate at 2... (gitlab-com/gl-infra/production#23085 - closed)
2026-10-01: The ci_runner_jobs SLI of the ci-ru... (gitlab-com/gl-infra/production#23077 - closed)
2026-10-01: gitlab-org/gitlab merge train is sl... (gitlab-com/gl-infra/production#23075 - closed)
2026-09-30: workhorse apdex SLO violation in cn... (gitlab-com/gl-infra/production#23069)
2026-10-01: websocketsServices error rate at 16... (gitlab-com/gl-infra/production#23063 - closed)
2026-10-01: data-server-rebuild-ansible-failed-... (gitlab-com/gl-infra/production#23062)
2026-09-30: The ci_runner_jobs SLI of the ci-ru... (gitlab-com/gl-infra/production#23058 - closed)
2026-09-30: Duo Code Review incorrectly formatt... (gitlab-com/gl-infra/production#23054 - closed)
2026-09-28: PGBouncer Client Connections per Pr... (gitlab-com/gl-infra/production#23033 - closed)
2026-09-28: The ci_runner_jobs SLI of the ci-ru... (gitlab-com/gl-infra/production#23032 - closed)

Instructions

  • Review the Additional incidents list and add the Deploys-blocked and RootCause labels if required.
  • Retrigger this pipeline job to update this report and metrics with the updated values. Consider this will discard any manual changes added.
  • Update the "weekly overview" table of this issue to also include:
    • any blocking CRs
  • Update the Deployments metric review epic.
    • Add a new row to the Overview section: Copy and paste the information from the Overview section in this issue and link to this issue in the Breakdown of blockers column.
    • Update the Graph: Update the data on the spreadsheet and then update the graph on the Deployments metric review epic.

📣 The deployment blockers per week can now be visualized in the Deployment Blockers Dashboard.

Edited by GitLab Release Tools Bot