Deployment Blockers - Week: 2026-04-27-2026-05-03

Overview

Start date	End date	Production deployments blocked for	# of blockers
2026-04-27	2026-05-03	37.0	50

Blocker types overview

Blocker Type	Time blocked on gstg	Time blocked on gprd
Change-request	3.0	3.0
RootCauseFlaky-Test	8.0	8.0
RootCauseDB-Migration	7.0	7.0
RootCauseSaturation	2.0	2.0
RootCauseConfig-Change	8.0	8.0
RootCauseExternal-Dependency	2.0	2.0
RootCauseSoftware-Change	7.0	7.0
Total	37.0	37.0

Weekly overview

Issue	Blocker type	gstg	gprd
FY27-Q1 HAProxy DR Gameday (outage in us-east1-c) (gitlab-com/gl-infra/production#21452 - closed)	Change-request	3.0	3.0
2026-04-27: Prompt Caching is not working on Ag... (gitlab-com/gl-infra/production#21893 - closed)		0.0	0.0
QA failure: qa/specs/features/browser_ui/10_sof... (#25227 - closed)	RootCauseFlaky-Test	0.5	0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/25228+		0.0	0.0
2026-04-27: Prod Post Deploy Migration timeout (gitlab-com/gl-infra/production#21894 - closed)	RootCauseDB-Migration	0.5	0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/25245+		0.0	0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/25253+		0.0	0.0
QA failure: qa/specs/features/browser_ui/3_crea... (#25254 - closed)	RootCauseFlaky-Test	0.5	0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/25255+		0.0	0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/25258+		0.0	0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/25262+		0.0	0.0
QA failure: qa/specs/features/browser_ui/10_sof... (#25269 - closed)	RootCauseFlaky-Test	0.5	0.5
QA failure: (#25281 - closed)	RootCauseFlaky-Test	0.5	0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/25283+		0.0	0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/25284+		0.0	0.0
2026-04-28: Disable ci-jobs-api backend ingress (gitlab-com/gl-infra/production#21903 - closed)	Change-request	0.0	0.0
2026-04-28: Prod post deploy migration failed (gitlab-com/gl-infra/production#21904 - closed)	RootCauseDB-Migration	1.0	1.0
2026-04-28: parsing error in the vulnerability ... (gitlab-com/gl-infra/production#21905 - closed)	RootCauseDB-Migration	2.0	2.0
2026-04-28: Sidekiq queueing SLO violation on m... (gitlab-com/gl-infra/production#21908 - closed)	RootCauseSaturation	2.0	2.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/25304+		0.0	0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/25311+		0.0	0.0
QA failure: qa/specs/features/browser_ui/2_plan... (#25321 - closed)	RootCauseFlaky-Test	0.5	0.5
QA failure: qa/specs/features/api/10_software_s... (#25326 - closed)	RootCauseFlaky-Test	0.5	0.5
2026-04-29: Prod post deploy migration failed a... (gitlab-com/gl-infra/production#21919 - closed)	RootCauseDB-Migration	0.5	0.5
QA failure: qa/specs/features/api/10_software_s... (#25358 - closed)	RootCauseFlaky-Test	0.5	0.5
QA failures on gstg-cny (#25375 - closed)	RootCauseFlaky-Test	0.5	0.5
QA failure: qa/specs/features/browser_ui/9_tena... (#25381 - closed)	RootCauseFlaky-Test	0.5	0.5
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21924+		0.0	0.0
QA failure: (#25392 - closed)	RootCauseFlaky-Test	0.5	0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/25409+		0.0	0.0
QA failure: (#25421 - closed)	RootCauseFlaky-Test	0.5	0.5
QA failures on gstg-cny (#25425 - closed)	RootCauseFlaky-Test	0.5	0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/25430+		0.0	0.0
2026-04-30: gitlab-assets-ee is not built becau... (gitlab-com/gl-infra/production#21933 - closed)	RootCauseConfig-Change	8.0	8.0
2026-04-30: The ci_runner_jobs SLI of the ci-ru... (gitlab-com/gl-infra/production#21936 - closed)	RootCauseExternal-Dependency	2.0	2.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/25460+		0.0	0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/25461+		0.0	0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/25464+		0.0	0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/25466+		0.0	0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/25467+		0.0	0.0
QA failures on gstg-cny (#25480 - closed)	RootCauseFlaky-Test	0.5	0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/25500+		0.0	0.0
QA failures on gstg-cny (#25523 - closed)	RootCauseFlaky-Test	0.5	0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#25535 - closed)	RootCauseFlaky-Test	0.5	0.5
2026-05-01: Block deployments to avoid achievem... (gitlab-com/gl-infra/production#21945 - closed)	RootCauseSoftware-Change	1.0	1.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/25541+		0.0	0.0
2026-05-01: gstg-cny-deployment-failing-due-to-... (gitlab-com/gl-infra/production#21950 - closed)	RootCauseSoftware-Change	6.0	6.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/25547+		0.0	0.0
2026-05-01: Migration failed to be executed on ... (gitlab-com/gl-infra/production#21955 - closed)	RootCauseDB-Migration	3.0	3.0
QA failure: qa/specs/features/browser_ui/3_crea... (#25558 - closed)	RootCauseFlaky-Test	0.5	0.5
Total		37.0	37.0

Additional incidents

Below is a list of production incidents created last week.

Click to expand

Issue
2026-05-03: Blackbox probe availability for htt... (gitlab-com/gl-infra/production#21961 - closed)
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21960+
2026-05-02: Orbit Prd NATS is down - [ERR] JetS... (gitlab-com/gl-infra/production#21959 - closed)
2026-05-02: Disk space on gitaly nodes predicte... (gitlab-com/gl-infra/production#21958 - closed)
2026-05-02: Sidekiq queueing apdex SLO violatio... (gitlab-com/gl-infra/production#21957 - closed)
2026-05-02: Elasticsearch production logging cl... (gitlab-com/gl-infra/production#21956 - closed)
2026-05-01: Sidekiq job queueing duration SLO v... (gitlab-com/gl-infra/production#21954 - closed)
2026-05-01: Disk space for gitaly nodes predict... (gitlab-com/gl-infra/production#21953 - closed)
2026-05-01: Chef/Cinc converges broken due to u... (gitlab-com/gl-infra/production#21952 - closed)
2026-05-01: At least 10% of chef-runs are faili... (gitlab-com/gl-infra/production#21947 - closed)
2026-05-01: Persistent volume claim storage-mim... (gitlab-com/gl-infra/production#21944 - closed)
2026-04-30: 2026-04-30 | Replica Pipeline for C... (gitlab-com/gl-infra/production#21938 - closed)
2026-04-30: Disk space utilization on gitaly no... (gitlab-com/gl-infra/production#21935 - closed)
2026-04-30: Sidekiq execution SLI apdex SLO vio... (gitlab-com/gl-infra/production#21934 - closed)
2026-04-30: The Disk inode Utilization per Devi... (gitlab-com/gl-infra/production#21932 - closed)
2026-04-29: Blackbox probe availability below 7... (gitlab-com/gl-infra/production#21929 - closed)
2026-04-29: Grafana datasource requests error r... (gitlab-com/gl-infra/production#21922 - closed)
2026-04-29: Unusual db_duration_s in logs (gitlab-com/gl-infra/production#21921 - closed)
2026-04-29: Sidekiq queueing SLO violation on m... (gitlab-com/gl-infra/production#21920 - closed)
2026-04-29: High transaction rollback rate on p... (gitlab-com/gl-infra/production#21915 - closed)
2026-04-28: Disk space utilization in gitaly ma... (gitlab-com/gl-infra/production#21914 - closed)
2026-04-28: High transaction rollback rate on P... (gitlab-com/gl-infra/production#21913 - closed)
2026-04-28: Grafana datasource requests to mimi... (gitlab-com/gl-infra/production#21911 - closed)
2026-04-28: 2026-04-28 | CI replica rebuild fai... (gitlab-com/gl-infra/production#21910 - closed)
2026-04-28: Security Report Ingestion Failing o... (gitlab-com/gl-infra/production#21907)
2026-04-27: 500 errors when loading gitlab-org/... (gitlab-com/gl-infra/production#21897 - closed)
2026-04-27: Sidekiq queueing SLO violation on m... (gitlab-com/gl-infra/production#21895 - closed)

Instructions

• Review the Additional incidents list and add the Deploys-blocked and RootCause labels if required.
• Retrigger this pipeline job to update this report and metrics with the updated values. Consider this will discard any manual changes added.
• Update the "weekly overview" table of this issue to also include:
  • any blocking CRs
• Update the Deployments metric review epic.
  • Add a new row to the Overview section: Copy and paste the information from the Overview section in this issue and link to this issue in the Breakdown of blockers column.
  • Update the Graph: Update the data on the spreadsheet and then update the graph on the Deployments metric review epic.

📣 The deployment blockers per week can now be visualized in the Deployment Blockers Dashboard.