Deployment Blockers - Week: 2025-12-01-2025-12-07

Overview

Start date End date Production deployments blocked for # of blockers
2025-12-01 2025-12-07 32.0 37

Blocker types overview

Blocker Type Time blocked on gstg Time blocked on gprd
Change-request 14.0 14.0
RootCauseFlaky-Test 10.5 10.5
RootCauseDB-Migration 1.0 1.0
RootCauseExternal-Dependency 3.0 3.0
Total 28.5 28.5

Weekly overview

Issue Blocker type gstg gprd
2025-12-01: Test Rollout of Ruby 3.3 to gstg-cn... (gitlab-com/gl-infra/production#20888 - closed) Change-request 4.0 4.0
QA failure: qa/specs/features/api/3_create/merg... (#22482 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/api/4_verify/file... (#22486 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/ee/browser_ui/18_... (#22489 - closed) RootCauseFlaky-Test 0.5 0.5
2025-12-01: Merged tab on the MR homepage times... (gitlab-com/gl-infra/production#20914 - closed) 3.0 3.0
QA failure: qa/specs/features/browser_ui/10_sof... (#22500 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/api/4_verify/file... (#22502 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#22506 - closed) 0.0 0.0
QA failure: qa/specs/features/browser_ui/4_veri... (#22508 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/22512+ 0.0 0.0
2025-12-03: Test Rollout of Ruby 3.3 to gstg-cn... (gitlab-com/gl-infra/production#20918 - closed) Change-request 10.0 10.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/22520+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/22523+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/22524+ 0.0 0.0
QA failure: qa/specs/features/api/4_verify/file... (#22527 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#22530 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20925+ 0.0 0.0
QA failure: qa/specs/features/api/4_verify/file... (#22550 - closed) 0.0 0.0
QA failure: qa/specs/features/browser_ui/3_crea... (#22560 - closed) RootCauseFlaky-Test 0.5 0.5
2025-12-04: gprd post-deploy migration failed (gitlab-com/gl-infra/production#20927 - closed) RootCauseDB-Migration 1.0 1.0
QA failure: qa/specs/features/browser_ui/3_crea... (#22572 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/api/4_verify/file... (#22595 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/api/3_create/merg... (#22599 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#22602 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/api/4_verify/file... (#22603 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/4_veri... (#22605 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#22607 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/api/4_verify/file... (#22609 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20939+ 0.5 0.5
QA failure: qa/specs/features/ee/browser_ui/2_p... (#22612 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/10_sof... (#22615 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/22616+ 0.0 0.0
QA failure: (#22618 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/4_veri... (#22620 - closed) 0.0 0.0
QA failures on gstg-cny (#22621 - closed) RootCauseExternal-Dependency 3.0 3.0
QA failures on gstg-cny (#22622 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20954+ 0.0 0.0
Total 32.0 32.0

Additional incidents

Below is a list of production incidents created last week.

Click to expand
Issue
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20953+
2025-12-07: Patroni processes down on multiple ... (gitlab-com/gl-infra/production#20952 - closed)
2025-12-06: Sidekiq queueing SLO violation on m... (gitlab-com/gl-infra/production#20951 - closed)
2025-12-06: rails_request error rate in ai-assi... (gitlab-com/gl-infra/production#20949 - closed)
2025-12-06: rails_request error rate SLO violat... (gitlab-com/gl-infra/production#20948 - closed)
2025-12-06: Sidekiq queueing SLI violation on m... (gitlab-com/gl-infra/production#20947 - closed)
2025-12-05: Sidekiq job execution error rate SL... (gitlab-com/gl-infra/production#20946 - closed)
2025-12-05: Prometheus unable to scrape exporte... (gitlab-com/gl-infra/production#20945 - closed)
2025-12-05: Gitlab Search API isn't available f... (gitlab-com/gl-infra/production#20943 - closed)
2025-12-05: Sidekiq queueing SLI apdex SLO viol... (gitlab-com/gl-infra/production#20941 - closed)
2025-12-05: At least 10% of chef-runs failing i... (gitlab-com/gl-infra/production#20937 - closed)
2025-12-05: High Database Replication Lag on gstg (gitlab-com/gl-infra/production#20936 - closed)
2025-12-05: Scheduled gitlab-production-snapsho... (gitlab-com/gl-infra/production#20934 - closed)
2025-12-04: Blackbox probes failing for https:/... (gitlab-com/gl-infra/production#20933 - closed)
2025-12-04: race condition is causing failing p... (gitlab-com/gl-infra/production#20929 - closed)
2025-12-04: Chef Converge Failures for Runner M... (gitlab-com/gl-infra/production#20928 - closed)
2025-12-04: rails_request error rate in ai-assi... (gitlab-com/gl-infra/production#20926 - closed)
2025-12-03: At least 10% of chef-runs are faili... (gitlab-com/gl-infra/production#20922 - closed)
2025-12-02: Error rate SLO violation on git ser... (gitlab-com/gl-infra/production#20921 - closed)
2025-12-02: Increased request queuing in API du... (gitlab-com/gl-infra/production#20920 - closed)
2025-12-02: PGBouncer client connections per pr... (gitlab-com/gl-infra/production#20919 - closed)
2025-12-02: Sidekiq jobs on shard catchall-b no... (gitlab-com/gl-infra/production#20917 - closed)
2025-12-01: Sidekiq queueing SLO violation on c... (gitlab-com/gl-infra/production#20916 - closed)

Instructions

  • Review the Additional incidents list and add the Deploys-blocked and RootCause labels if required.
  • Retrigger this pipeline job to update this report and metrics with the updated values. Consider this will discard any manual changes added.
  • Update the "weekly overview" table of this issue to also include:
    • any blocking CRs
  • Update the Deployments metric review epic.
    • Add a new row to the Overview section: Copy and paste the information from the Overview section in this issue and link to this issue in the Breakdown of blockers column.
    • Update the Graph: Update the data on the spreadsheet and then update the graph on the Deployments metric review epic.

📣 The deployment blockers per week can now be visualized in the Deployment Blockers Dashboard.

Edited by John Skarbek