Deployment Blockers - Week: 2026-05-18-2026-05-24

Overview

Start date End date Production deployments blocked for # of blockers
2026-05-18 2026-05-24 16.5 50

Blocker types overview

Blocker Type Time blocked on gstg Time blocked on gprd
RootCauseSaturation 7.0 7.0
RootCauseFlaky-Test 7.5 7.5
RootCauseMalicious-Traffic 2.0 2.0
Total 16.5 16.5

Weekly overview

Issue Blocker type gstg gprd
2026-05-19: Redis primary CPU saturation on red... (gitlab-com/gl-infra/production#22163 - closed) RootCauseSaturation 5.0 5.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22182+ RootCauseMalicious-Traffic 2.0 2.0
2026-05-18: Jobs not being picked up/stuck pend... (gitlab-com/gl-infra/production#22145 - closed) RootCauseSaturation 2.0 2.0
QA failure: (#26399 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/9_tena... (#26536 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/2_plan... (#26379 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/9_tena... (#26543 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/9_tena... (#26549 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#26562 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#26402 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/9_tena... (#26403 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/9_tena... (#26410 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/api/10_software_s... (#26533 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/2_plan... (#26418 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#26339 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#26458 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/api/10_software_s... (#26327 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/8_moni... (#26326 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/26513+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26507+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26465+ 0.0 0.0
QA failure: qa/specs/features/browser_ui/3_crea... (#26566 - closed) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26583+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26589+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26590+ 0.0 0.0
2026-05-22: Workhorse web requests in us-east1-... (gitlab-com/gl-infra/production#22194 - closed) RootCauseSoftware-Change 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26595+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26598+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26600+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26610+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26612+ 0.0 0.0
[GPRD] Retry: Route canary traffic away from NGINX (gitlab-com/gl-infra/production#22055 - closed) Change-request 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26311+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26330+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26334+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26345+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26347+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26348+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26349+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26350+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26353+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26358+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26375+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26377+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26389+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26414+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26424+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26456+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26459+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26460+ 0.0 0.0
Total 16.5 16.5

Additional incidents

Below is a list of production incidents created last week.

Click to expand
Issue
2026-05-24: GraphQL query error rate violating ... (gitlab-com/gl-infra/production#22200)
2026-05-24: GraphQL query error rate violating ... (gitlab-com/gl-infra/production#22199 - closed)
2026-05-23: Sidekiq job queueing duration SLO v... (gitlab-com/gl-infra/production#22198 - closed)
2026-05-23: Apdex SLO violation for server_code... (gitlab-com/gl-infra/production#22197 - closed)
2026-05-23: GraphQL query error rate violating ... (gitlab-com/gl-infra/production#22196 - closed)
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22195+
2026-05-22: No traffic reported by http-router ... (gitlab-com/gl-infra/production#22193 - closed)
2026-05-22: Apdex SLO violation for code comple... (gitlab-com/gl-infra/production#22189 - closed)
2026-05-22: The Disk inode Utilization per Devi... (gitlab-com/gl-infra/production#22188 - closed)
2026-05-21: GCP quota utilization for T2D_CPUS ... (gitlab-com/gl-infra/production#22187 - closed)
2026-05-21: Delayed CI job output due to rails_... (gitlab-com/gl-infra/production#22181 - closed)
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22179+
2026-05-20: Apdex SLO violation for server_code... (gitlab-com/gl-infra/production#22178 - closed)
2026-05-20: CustomersDot Production - CloudSQL ... (gitlab-com/gl-infra/production#22177 - closed)
2026-05-20: Web requests through workhorse belo... (gitlab-com/gl-infra/production#22175 - closed)
2026-05-20: Rails logs not available in Kibana (gitlab-com/gl-infra/production#22173 - closed)
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22168+
2026-05-19: gitlab-pages tests failing with X50... (gitlab-com/gl-infra/production#22166 - closed)
2026-05-19: Apdex SLO violation for web request... (gitlab-com/gl-infra/production#22165 - closed)
2026-05-19: workhorse_auth_api apdex SLO violat... (gitlab-com/gl-infra/production#22162 - closed)
2026-05-19: Apdex SLO violation for web service... (gitlab-com/gl-infra/production#22161)
2026-05-19: Sidekiq job queueing apdex SLO viol... (gitlab-com/gl-infra/production#22160 - closed)
2026-05-19: 2026-05-19: Data-Server Rebuild Ans... (gitlab-com/gl-infra/production#22157 - closed)
2026-05-19: Cluster autoscaler scaleup error ra... (gitlab-com/gl-infra/production#22156 - closed)
2026-05-19: Registry loadbalancer error rate vi... (gitlab-com/gl-infra/production#22155 - closed)
2026-05-19: No traffic reported by nginx for cu... (gitlab-com/gl-infra/production#22154 - closed)
2026-05-18: GKE cluster scale-up fails intermit... (gitlab-com/gl-infra/production#22151 - closed)
2026-05-18: Cluster autoscaler scaleup error ra... (gitlab-com/gl-infra/production#22150 - closed)
2026-05-18: Serval Email Notification stuck in ... (gitlab-com/gl-infra/production#22149 - closed)
2026-05-18: Cluster autoscaler scaleup error ra... (gitlab-com/gl-infra/production#22147 - closed)
2026-05-18: The ci_runner_jobs SLI of the ci-ru... (gitlab-com/gl-infra/production#22146)
2026-05-18: Cluster autoscaler scaleup error ra... (gitlab-com/gl-infra/production#22142 - closed)

Instructions

  • Review the Additional incidents list and add the Deploys-blocked and RootCause labels if required.
  • Retrigger this pipeline job to update this report and metrics with the updated values. Consider this will discard any manual changes added.
  • Update the "weekly overview" table of this issue to also include:
    • any blocking CRs
  • Update the Deployments metric review epic.
    • Add a new row to the Overview section: Copy and paste the information from the Overview section in this issue and link to this issue in the Breakdown of blockers column.
    • Update the Graph: Update the data on the spreadsheet and then update the graph on the Deployments metric review epic.

📣 The deployment blockers per week can now be visualized in the Deployment Blockers Dashboard.

Edited by Mawreen Dela Cruz