Deployment Blockers - Week: 2026-05-04-2026-05-10

Overview

Start date End date Production deployments blocked for # of blockers
2026-05-04 2026-05-10 60.5 61

Blocker types overview

Blocker Type Time blocked on gstg Time blocked on gprd
Change-request 5.0 5.0
RootCauseSoftware-Change 10.0 10.0
RootCauseDB-Migration 11.0 11.0
RootCauseConfig-Change 8.5 8.5
RootCauseSaturation 7.0 7.0
RootCauseFlaky-Test 19.0 19.0
Total 60.5 60.5

Weekly overview

Issue Blocker type gstg gprd
[GPRD] Route canary traffic away from NGINX (gitlab-com/gl-infra/production#21809 - closed) Change-request 5.0 5.0
2026-05-04: Gitaly error: maximum time in concu... (gitlab-com/gl-infra/production#21964 - closed) 0.0 0.0
Track master broken incident blocking time (#25579 - closed) RootCauseSoftware-Change 5.0 5.0
2026-05-04: Production post deploy migration fa... (gitlab-com/gl-infra/production#21970 - closed) RootCauseDB-Migration 2.0 2.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/25590+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/25591+ 0.0 0.0
2026-05-04: Production post deploy migration fa... (gitlab-com/gl-infra/production#21973 - closed) RootCauseDB-Migration 1.0 1.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/25593+ 0.0 0.0
2026-05-04: Production post deploy migration fa... (gitlab-com/gl-infra/production#21974 - closed) RootCauseDB-Migration 1.0 1.0
2026-05-04: GITLAB_DEPENCY_TOKEN fails to authe... (gitlab-com/gl-infra/production#21976 - closed) RootCauseConfig-Change 0.5 0.5
2026-05-04: K8s deployment is stuck (gitlab-com/gl-infra/production#21978 - closed) RootCauseSaturation 1.0 1.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/25610+ 0.0 0.0
QA failure: qa/specs/features/browser_ui/4_veri... (#25614 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/25631+ 0.0 0.0
QA failure: qa/specs/features/ee/api/2_plan/epi... (#25656 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/25657+ 0.0 0.0
2026-05-05: Production post deploy migration fa... (gitlab-com/gl-infra/production#21988 - closed) RootCauseDB-Migration 3.0 3.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/25667+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/25669+ 0.0 0.0
QA failures on gstg-cny (#25675 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21989+ RootCauseConfig-Change 4.0 4.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/25692+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/25694+ 0.0 0.0
QA failure: qa/specs/features/browser_ui/3_crea... (#25700 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#25730 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/25735+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/25736+ 0.0 0.0
QA failure: qa/specs/features/browser_ui/2_plan... (#25739 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: (#25741 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/25743+ 0.0 0.0
QA failure: (#25747 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: (#25753 - closed) RootCauseFlaky-Test 0.5 0.5
2026-05-06: staging qa-smoke test failing with ... (gitlab-com/gl-infra/production#22003 - closed) RootCauseConfig-Change 1.0 1.0
QA failure: qa/specs/features/browser_ui/3_crea... (#25762 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#25770 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: (#25783 - closed) RootCauseFlaky-Test 0.5 0.5
2026-05-06: Loadbalancer error rate for api ser... (gitlab-com/gl-infra/production#22008 - closed) RootCauseConfig-Change 3.0 3.0
QA failure: qa/specs/features/browser_ui/3_crea... (#25843 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/api/3_create/merg... (#25850 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#25852 - closed) RootCauseFlaky-Test 0.5 0.5
qa-smoke tests in staging failing frequently du... (#25858 - closed) RootCauseFlaky-Test 6.0 6.0
QA failure: qa/specs/features/ee/browser_ui/18_... (#25864 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/25869+ 0.0 0.0
QA failure: qa/specs/features/ee/api/2_plan/epi... (#25871 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gprd-cny (#25873 - closed) RootCauseFlaky-Test 0.5 0.5
2026-05-07: Jobs duration is incorrect (gitlab-com/gl-infra/production#22016 - closed) RootCauseSaturation 6.0 6.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/25878+ 0.0 0.0
QA failures on gprd-cny (#25891 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/ee/browser_ui/2_p... (#25892 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/25905+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/25910+ 0.0 0.0
QA failure: qa/specs/features/ee/browser_ui/2_p... (#25958 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#25964 - closed) RootCauseFlaky-Test 0.5 0.5
2026-05-08: Prod Post Deploy Migration failed a... (gitlab-com/gl-infra/production#22022 - closed) RootCauseDB-Migration 4.0 4.0
2026-05-08: Fix pipeline flows fail due to miss... (gitlab-com/gl-infra/production#22023 - closed) 0.0 0.0
QA failure: qa/specs/features/browser_ui/3_crea... (#25986 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/2_plan... (#25996 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/9_tena... (#26003 - closed) RootCauseFlaky-Test 0.5 0.5
2026-05-08: Upgrading DUO_CLI_VERSION to "8.93.... (gitlab-com/gl-infra/production#22031 - closed) RootCauseSoftware-Change 5.0 5.0
QA failure: qa/specs/features/browser_ui/9_tena... (#26011 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/10_sof... (#26020 - closed) 0.0 0.0
Total 60.5 60.5

Additional incidents

Below is a list of production incidents created last week.

Click to expand
Issue
2026-05-10: Increased request latency due to Pu... (gitlab-com/gl-infra/production#22039 - closed)
2026-05-10: Elastic cluster gprd-indexing-20220... (gitlab-com/gl-infra/production#22038 - closed)
2026-05-10: Apdex SLO violation for rails_prima... (gitlab-com/gl-infra/production#22037 - closed)
2026-05-09: Disk space on gitaly nodes predicte... (gitlab-com/gl-infra/production#22036 - closed)
2026-05-09: Sidekiq queueing SLI apdex SLO viol... (gitlab-com/gl-infra/production#22035 - closed)
2026-05-09: Blackbox probe availability for htt... (gitlab-com/gl-infra/production#22034 - closed)
2026-05-08: Tier-3 pipelines are not running an... (gitlab-com/gl-infra/production#22030 - closed)
2026-05-08: Sidekiq queueing SLO violation on m... (gitlab-com/gl-infra/production#22029 - closed)
2026-05-08: GCP quota usage near capacity for m... (gitlab-com/gl-infra/production#22027 - closed)
2026-05-08: Grafana dashboard(s) erroring with ... (gitlab-com/gl-infra/production#22026 - closed)
2026-05-08: Blackbox probes failing for https:/... (gitlab-com/gl-infra/production#22025 - closed)
2026-05-07: External DNS records are stale (gitlab-com/gl-infra/production#22019 - closed)
2026-05-07: Sidekiq queueing SLO violation on u... (gitlab-com/gl-infra/production#22018)
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22017+
2026-05-07: Sidekiq queueing SLI apdex SLO viol... (gitlab-com/gl-infra/production#22015)
2026-05-07: SSL certificate for workspaces.gitl... (gitlab-com/gl-infra/production#22013 - closed)
2026-05-06: Sidekiq queueing SLO violation on l... (gitlab-com/gl-infra/production#22012 - closed)
2026-05-06: Grafana datasource requests to mimi... (gitlab-com/gl-infra/production#22011 - closed)
2026-05-06: Blackbox probe availability to http... (gitlab-com/gl-infra/production#22009 - closed)
2026-05-06: GCP T2D_CPUS quota saturation in us... (gitlab-com/gl-infra/production#22007 - closed)
2026-05-06: Cluster autoscaler scaleup error ra... (gitlab-com/gl-infra/production#22006)
2026-05-06: Sidekiq queueing apdex SLO violatio... (gitlab-com/gl-infra/production#22005 - closed)
2026-05-05: Sidekiq job queueing SLO violated o... (gitlab-com/gl-infra/production#21998 - closed)
2026-05-05: Sidekiq queueing SLI SLO violation ... (gitlab-com/gl-infra/production#21997 - closed)
2026-05-05: Frequent GKE node scale-up errors d... (gitlab-com/gl-infra/production#21996 - closed)
2026-05-05: Cluster autoscaler scaleup error ra... (gitlab-com/gl-infra/production#21995 - closed)
2026-05-05: Low-urgency Sidekiq PgBouncer satur... (gitlab-com/gl-infra/production#21993 - closed)
2026-05-05: Increased transaction rollback rate... (gitlab-com/gl-infra/production#21992 - closed)
2026-05-05: High transaction rollback rate on p... (gitlab-com/gl-infra/production#21991 - closed)
2026-05-05: High transaction rollback rate on p... (gitlab-com/gl-infra/production#21986 - closed)
2026-05-05: Sidekiq queueing SLO violation on c... (gitlab-com/gl-infra/production#21985 - closed)
2026-05-04: GCP scheduled snapshots missing for... (gitlab-com/gl-infra/production#21984 - closed)
2026-05-04: Cluster autoscaler scaleup error ra... (gitlab-com/gl-infra/production#21982 - closed)
2026-05-04: 2026-05-04 | CI replica failed on r... (gitlab-com/gl-infra/production#21980 - closed)
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21979+
2026-05-04: Sidekiq queueing duration SLO viola... (gitlab-com/gl-infra/production#21977 - closed)
2026-05-04: No traffic reported by primary_serv... (gitlab-com/gl-infra/production#21972 - closed)
2026-05-04: Error rate of 76.38% on atlantis_go... (gitlab-com/gl-infra/production#21965 - closed)

Instructions

  • Review the Additional incidents list and add the Deploys-blocked and RootCause labels if required.
  • Retrigger this pipeline job to update this report and metrics with the updated values. Consider this will discard any manual changes added.
  • Update the "weekly overview" table of this issue to also include:
    • any blocking CRs
  • Update the Deployments metric review epic.
    • Add a new row to the Overview section: Copy and paste the information from the Overview section in this issue and link to this issue in the Breakdown of blockers column.
    • Update the Graph: Update the data on the spreadsheet and then update the graph on the Deployments metric review epic.

📣 The deployment blockers per week can now be visualized in the Deployment Blockers Dashboard.

Edited by Dat Tang