Deployment Blockers - Week: 2025-10-06-2025-10-12

Overview

Start date End date Production deployments blocked for # of blockers
2025-10-06 2025-10-12 10.0 34

Blocker types overview

Blocker Type Time blocked on gstg Time blocked on gprd
RootCauseSoftware-Change 1.0 1.0
RootCauseFlaky-Test 5.0 5.0
Total 6.0 6.0

Weekly overview

Issue Blocker type gstg gprd
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20688+ 2.0 2.0
QA failures on gstg-cny (#21586 - closed) RootCauseSoftware-Change 1.0 1.0
2025-10-07: The gitlab_sshd git service unavail... (gitlab-com/gl-infra/production#20680 - closed) 1.0 1.0
2025-10-07: Gitlab unavailable due to load (gitlab-com/gl-infra/production#20686 - closed) 1.0 1.0
QA failure: qa/specs/features/browser_ui/10_sof... (#21596 - closed) RootCauseFlaky-Test 1.0 1.0
QA failure: qa/specs/features/browser_ui/3_crea... (#21594 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gprd-cny (#21598 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/8_moni... (#21607 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gprd-cny (#21608 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#21629 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gprd-cny (#21636 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#21651 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/10_sof... (#21652 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20640+ Change-request 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/21585+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/21589+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/21590+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/21593+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/21595+ 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20681+ 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20683+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/21599+ 0.0 0.0
2025-10-07: 500 Error when attempting to create... (gitlab-com/gl-infra/production#20687 - closed) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/21602+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/21603+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/21604+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/21605+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/21610+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/21611+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/21613+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/21618+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/21624+ 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20705+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/21656+ 0.0 0.0
Total 10.0 10.0

Additional incidents

Below is a list of production incidents created last week.

Click to expand
Issue
2025-10-11: rails_request error rate for ai-ass... (gitlab-com/gl-infra/production#20711 - closed)
2025-10-10: Loadbalancer 5xx error rate violati... (gitlab-com/gl-infra/production#20710 - closed)
2025-10-10: Workhorse web requests apdex below ... (gitlab-com/gl-infra/production#20708 - closed)
2025-10-10: rails_request error rate for ai-ass... (gitlab-com/gl-infra/production#20706 - closed)
2025-10-09: Sidekiq queueing SLO violation on c... (gitlab-com/gl-infra/production#20704 - closed)
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20699+
2025-10-08: New UI doesn't show page number in ... (gitlab-com/gl-infra/production#20694 - closed)
2025-10-08: PubSub messages queuing on pubsub-r... (gitlab-com/gl-infra/production#20691 - closed)
2025-10-08: Sidekiq queueing apdex SLO violatio... (gitlab-com/gl-infra/production#20690 - closed)
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20684+
2025-10-07: Gitaly goserver apdex SLO violation... (gitlab-com/gl-infra/production#20677 - closed)
2025-10-07: Gitaly GRPC::DeadlineExceeded and G... (gitlab-com/gl-infra/production#20676)
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20674+
2025-10-06: PubSub messages queuing on pubsub-r... (gitlab-com/gl-infra/production#20673 - closed)
2025-10-06: Sidekiq queueing apdex SLO violatio... (gitlab-com/gl-infra/production#20672 - closed)
2025-10-06: Increased request queuing in api se... (gitlab-com/gl-infra/production#20671 - closed)
2025-10-06: Blackbox probes failing for https:/... (gitlab-com/gl-infra/production#20667 - closed)
2025-10-06: Requests queuing on internal-api in... (gitlab-com/gl-infra/production#20666 - closed)

Instructions

  • Review the Additional incidents list and add the Deploys-blocked and RootCause labels if required.
  • Retrigger this pipeline job to update this report and metrics with the updated values. Consider this will discard any manual changes added.
  • Update the "weekly overview" table of this issue to also include:
    • any blocking CRs
  • Update the Deployments metric review epic.
    • Add a new row to the Overview section: Copy and paste the information from the Overview section in this issue and link to this issue in the Breakdown of blockers column.
    • Update the Graph: Update the data on the spreadsheet and then update the graph on the Deployments metric review epic.

📣 The deployment blockers per week can now be visualized in the Deployment Blockers Dashboard.

Edited by Reuben Pereira