Deployment Blockers - Week: 2026-03-16-2026-03-22

Overview

Start date End date Production deployments blocked for # of blockers
2026-03-16 2026-03-22 23.5 27

Blocker types overview

Blocker Type Time blocked on gstg Time blocked on gprd
Change-request 4.0 4.0
RootCauseDB-Migration 2.0 2.0
RootCauseFlaky-Test 4.5 4.5
RootCauseSoftware-Change 13.0 13.0
Total 23.5 23.5

Weekly overview

Issue Blocker type gstg gprd
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/20872+ Change-request 2.0 2.0
2026-03-18: Enable workload identity and Signed... (gitlab-com/gl-infra/production#21494 - closed) Change-request 0.0 0.0
Enable Topology Service client for production G... (gitlab-com/gl-infra/production#21546 - closed) Change-request 2.0 2.0
2026-03-16: Post-deploy migration gprd failed a... (gitlab-com/gl-infra/production#21557 - closed) RootCauseDB-Migration 2.0 2.0
QA failure: (#24151 - closed) RootCauseFlaky-Test 0.5 0.5
2026-03-16: Database lock contention causing sl... (gitlab-com/gl-infra/production#21562 - closed) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/24174+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/24175+ 0.0 0.0
QA failure: qa/specs/features/browser_ui/3_crea... (#24181 - closed) RootCauseFlaky-Test 0.5 0.5
2026-03-17: 500 error when creating MR and API ... (gitlab-com/gl-infra/production#21573 - closed) RootCauseSoftware-Change 5.0 5.0
QA failure: qa/specs/features/browser_ui/10_sof... (#24202 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/24217+ 0.0 0.0
QA failure: (#24219 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/24220+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/24232+ 0.0 0.0
QA failure: qa/specs/features/ee/api/2_plan/epi... (#24236 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/10_sof... (#24237 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/10_sof... (#24244 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/api/10_software_s... (#24249 - closed) RootCauseFlaky-Test 0.5 0.5
2026-03-19: The loadbalancer SLI of the ci-runn... (gitlab-com/gl-infra/production#21597) 0.0 0.0
QA failure: qa/specs/features/browser_ui/2_plan... (#24261 - closed) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/24269+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/24270+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/24274+ 0.0 0.0
2026-03-19: gstg-cny deploy blocked - package n... (gitlab-com/gl-infra/production#21603 - closed) RootCauseSoftware-Change 8.0 8.0
QA failure: qa/specs/features/api/10_software_s... (#24287 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/24290+ 0.0 0.0
Total 23.5 23.5

Additional incidents

Below is a list of production incidents created last week.

Click to expand
Issue
2026-03-20: GitLab in general feels slow, MR da... (gitlab-com/gl-infra/production#21606 - closed)
2026-03-20: Customers affected by GitLab is cur... (gitlab-com/gl-infra/production#21605 - closed)
2026-03-19: External DNS records are stale in g... (gitlab-com/gl-infra/production#21602 - closed)
2026-03-19: Transaction rollback error rate exc... (gitlab-com/gl-infra/production#21601 - closed)
2026-03-19: Gitaly CNY showing signs of degrada... (gitlab-com/gl-infra/production#21600 - closed)
2026-03-19: Loadbalancer error rate for web ser... (gitlab-com/gl-infra/production#21596)
2026-03-19: Mimir recording rule failures (gitlab-com/gl-infra/production#21594 - closed)
2026-03-18: Grafana datasource requests from mo... (gitlab-com/gl-infra/production#21590 - closed)
2026-03-18: Loadbalancer 5xx error rate for web... (gitlab-com/gl-infra/production#21586 - closed)
2026-03-18: Unexpected Rate limits (gitlab-com/gl-infra/production#21584 - closed)
2026-03-18: atlantis_google_lb error rate at 3.... (gitlab-com/gl-infra/production#21582 - closed)
2026-03-18: packagecloud enterprise license exp... (gitlab-com/gl-infra/production#21580 - closed)
2026-03-17: Sidekiq job queueing SLO violation ... (gitlab-com/gl-infra/production#21577)
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21574+
2026-03-16: Grafana datasource error rate for m... (gitlab-com/gl-infra/production#21566 - closed)
2026-03-16: Sidekiq queueing SLI SLO violation ... (gitlab-com/gl-infra/production#21561 - closed)
2026-03-16: Gitlab Pages 404 error rate increas... (gitlab-com/gl-infra/production#21559 - closed)
2026-03-16: Sidekiq job queueing SLO violation ... (gitlab-com/gl-infra/production#21558 - closed)

Instructions

  • Review the Additional incidents list and add the Deploys-blocked and RootCause labels if required.
  • Retrigger this pipeline job to update this report and metrics with the updated values. Consider this will discard any manual changes added.
  • Update the "weekly overview" table of this issue to also include:
    • any blocking CRs
  • Update the Deployments metric review epic.
    • Add a new row to the Overview section: Copy and paste the information from the Overview section in this issue and link to this issue in the Breakdown of blockers column.
    • Update the Graph: Update the data on the spreadsheet and then update the graph on the Deployments metric review epic.

📣 The deployment blockers per week can now be visualized in the Deployment Blockers Dashboard.

Edited by GitLab Release Tools Bot