Deployment Blockers - Week: 2026-04-13-2026-04-19

Overview

Start date End date Production deployments blocked for # of blockers
2026-04-13 2026-04-19 26.5 20

Blocker types overview

Blocker Type Time blocked on gstg Time blocked on gprd
RootCauseFlaky-Test 3.5 3.5
Total 3.5 3.5

Weekly overview

Issue Blocker type gstg gprd
QA failure: qa/specs/features/browser_ui/4_veri... (#24671 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/ee/browser_ui/18_... (#24676 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/24681+ 0.0 0.0
QA failure: qa/specs/features/browser_ui/3_crea... (#24684 - closed) RootCauseFlaky-Test 0.5 0.5
2026-04-14: Customers getting "pull access deni... (gitlab-com/gl-infra/production#21778 - closed) 9.0 9.0
QA failures on gprd-cny (#24687 - closed) 0.0 0.0
QA failure: qa/specs/features/browser_ui/10_sof... (#24690 - closed) 0.0 0.0
QA failure: qa/specs/features/browser_ui/10_sof... (#24693 - closed) 0.0 0.0
QA failure: (#24701 - closed) 0.0 0.0
2026-04-15: CNG builds failing on gitlab-openbao (gitlab-com/gl-infra/production#21797 - closed) 14.0 14.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/24739+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/24763+ 0.0 0.0
QA failure: qa/specs/features/ee/browser_ui/2_p... (#24767 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/24777+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/24786+ 0.0 0.0
QA failure: qa/specs/features/ee/browser_ui/2_p... (#24830 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21819+ 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21820+ 0.0 0.0
QA failure: (#24884 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#24887 - closed) RootCauseFlaky-Test 0.5 0.5
Total 26.5 26.5

Additional incidents

Below is a list of production incidents created last week.

Click to expand
Issue
2026-04-19: Workhorse API authentication errors... (gitlab-com/gl-infra/production#21827 - closed)
2026-04-19: Loadbalancer error rate for web ser... (gitlab-com/gl-infra/production#21825 - closed)
2026-04-19: The ci_runner_jobs SLI of the ci-ru... (gitlab-com/gl-infra/production#21824 - closed)
2026-04-17: Apdex SLO violation for code comple... (gitlab-com/gl-infra/production#21823 - closed)
2026-04-17: The ci_runner_jobs SLI of the ci-ru... (gitlab-com/gl-infra/production#21816 - closed)
2026-04-17: Sidekiq queueing latency SLO violat... (gitlab-com/gl-infra/production#21815 - closed)
2026-04-17: Prometheus scrape failures on patro... (gitlab-com/gl-infra/production#21814 - closed)
2026-04-16: Increased transaction rollbacks on ... (gitlab-com/gl-infra/production#21812 - closed)
2026-04-16: Increased transaction rollbacks on ... (gitlab-com/gl-infra/production#21811 - closed)
2026-04-16: gitlab-org/gitlab all new pipelines... (gitlab-com/gl-infra/production#21810 - closed)
2026-04-16: At least 10% of chef-runs failing i... (gitlab-com/gl-infra/production#21807 - closed)
2026-04-15: 2026-04-15 Data-Server Rebuild Ansi... (gitlab-com/gl-infra/production#21805 - closed)
2026-04-15: PGBouncer client connections per pr... (gitlab-com/gl-infra/production#21800 - closed)
2026-04-14: test-infection-scanner job in RC pi... (gitlab-com/gl-infra/production#21787 - closed)
2026-04-14: AI Gateway deployments failing (gitlab-com/gl-infra/production#21786 - closed)
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21784+
2026-04-14: Loadbalancer error rate for web ser... (gitlab-com/gl-infra/production#21782 - closed)
2026-04-14: Apdex SLO violation for SQL transac... (gitlab-com/gl-infra/production#21781 - closed)
2026-04-13: Rails SQL transactions to Postgres ... (gitlab-com/gl-infra/production#21772 - closed)
2026-04-13: Apdex SLO violation for SQL transac... (gitlab-com/gl-infra/production#21770 - closed)
2026-04-13: HTTP 500 error on https://gitlab.co... (gitlab-com/gl-infra/production#21769 - closed)
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21767+

Instructions

  • Review the Additional incidents list and add the Deploys-blocked and RootCause labels if required.
  • Retrigger this pipeline job to update this report and metrics with the updated values. Consider this will discard any manual changes added.
  • Update the "weekly overview" table of this issue to also include:
    • any blocking CRs
  • Update the Deployments metric review epic.
    • Add a new row to the Overview section: Copy and paste the information from the Overview section in this issue and link to this issue in the Breakdown of blockers column.
    • Update the Graph: Update the data on the spreadsheet and then update the graph on the Deployments metric review epic.

📣 The deployment blockers per week can now be visualized in the Deployment Blockers Dashboard.

Edited by Reuben Pereira