Deployment Blockers - Week: 2026-08-03-2026-08-09

Overview

Start date End date Production deployments blocked for # of blockers
2026-08-03 2026-08-09 34.0 44

Blocker types overview

Blocker Type Time blocked on gstg Time blocked on gprd
RootCauseDB-Migration 6.0 6.0
RootCauseFlaky-Test 7.0 7.0
RootCauseSaturation 7.0 7.0
RootCauseSoftware-Change 11.0 11.0
~"RootCause::ExternalAgentMaliciousBehavior" 3.0 3.0
Total 34.0 34.0

Weekly overview

Issue Blocker type gstg gprd
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22659+ RootCauseSoftware-Change 10.0 10.0
2026-08-03: Loadbalancer SLI error rate at 0.16... (gitlab-com/gl-infra/production#22625 - closed) RootCauseSaturation 7.0 7.0
2026-08-05: Wraparound prevention vacuum in Pos... (gitlab-com/gl-infra/production#22642 - closed) RootCauseDB-Migration 5.0 5.0
2026-08-06: sidekiq_queueing apdex 70.08% on lo... (gitlab-com/gl-infra/production#22661 - closed) ~"RootCause::ExternalAgentMaliciousBehavior" 3.0 3.0
2026-08-03: gprd post-deploy migration failed w... (gitlab-com/gl-infra/production#22622 - closed) RootCauseDB-Migration 1.0 1.0
2026-08-04: Duo code review consistently failing (gitlab-com/gl-infra/production#22633 - closed) RootCauseSoftware-Change 1.0 1.0
QA failure: qa/specs/features/browser_ui/3_crea... (#27613 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gprd-cny (#27615 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/ee/api/2_plan/epi... (#27605 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/2_plan... (#27618 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gprd-cny (#27601 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/2_plan... (#27621 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gprd-cny (#27599 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/ee/browser_ui/18_... (#27598 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/ee/api/2_plan/epi... (#27597 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/api/10_software_s... (#27596 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: (#27594 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: (#27593 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: (#27592 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/9_tena... (#27587 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/27620+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27619+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27622+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27617+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27616+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27623+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27624+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27625+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27614+ 0.0 0.0
QA failures on gprd-cny (#27612 - closed) 0.0 0.0
2026-08-05: GitLabDuo not responding and CI str... (gitlab-com/gl-infra/production#22655) 0.0 0.0
QA failures on gprd-cny (#27611 - closed) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27610+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27609+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27608+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27606+ 0.0 0.0
QA failures on gprd-cny (#27604 - closed) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27603+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27602+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27600+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27591+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27590+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27589+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27588+ 0.0 0.0
Total 34.0 34.0

Additional incidents

Below is a list of production incidents created last week.

Click to expand
Issue
2026-08-07: http-router worker_requests traffic... (gitlab-com/gl-infra/production#22669 - closed)
2026-08-06: Sidekiq queueing apdex on quarantin... (gitlab-com/gl-infra/production#22665 - closed)
2026-08-06: Production Canary cannot run Migrat... (gitlab-com/gl-infra/production#22662 - closed)
2026-08-06: Disk space utilization on patroni m... (gitlab-com/gl-infra/production#22660 - closed)
2026-08-05: Sidekiq queueing apdex at 55.62% on... (gitlab-com/gl-infra/production#22654 - closed)
2026-08-05: Increased rate of usage billing blo... (gitlab-com/gl-infra/production#22653 - closed)
2026-08-05: server_code_completions apdex at 69... (gitlab-com/gl-infra/production#22651 - closed)
2026-08-05: Sidekiq queueing apdex at 63.45% on... (gitlab-com/gl-infra/production#22650 - closed)
2026-08-05: Failure in getting subscription usa... (gitlab-com/gl-infra/production#22649 - closed)
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22648+
2026-08-05: sidekiq_execution apdex 93.94% on u... (gitlab-com/gl-infra/production#22645 - closed)
2026-08-05: Data-Server Rebuild Ansible fails a... (gitlab-com/gl-infra/production#22644 - closed)
2026-08-05: Missing Manifests for Multiple Rece... (gitlab-com/gl-infra/production#22643 - closed)
2026-08-04: Pods stuck Unschedulable in gprd-us... (gitlab-com/gl-infra/production#22640 - closed)
2026-08-05: 3 pods unschedulable in gprd-us-eas... (gitlab-com/gl-infra/production#22639)
2026-08-04: mimir_query_scheduler and mimir_sto... (gitlab-com/gl-infra/production#22638 - closed)
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22637+
2026-08-04: QA failing in Production Canary (gitlab-com/gl-infra/production#22636 - closed)
2026-08-04: Postgres int4 ID saturation on merg... (gitlab-com/gl-infra/production#22635 - closed)
2026-08-04: Websockets loadbalancer error rate ... (gitlab-com/gl-infra/production#22632 - closed)
2026-08-04: The Disk inode Utilization per Devi... (gitlab-com/gl-infra/production#22630 - closed)
2026-08-04: The polling SLI of the ci-runners s... (gitlab-com/gl-infra/production#22629 - closed)
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22628+
2026-08-04: web loadbalancer error rate violati... (gitlab-com/gl-infra/production#22627 - closed)
2026-08-03: server_code_completions apdex at 69... (gitlab-com/gl-infra/production#22626 - closed)
2026-08-03: Workhorse apdex SLO violation at 98... (gitlab-com/gl-infra/production#22623 - closed)

Instructions

  • Review the Additional incidents list and add the Deploys-blocked and RootCause labels if required.
  • Retrigger this pipeline job to update this report and metrics with the updated values. Consider this will discard any manual changes added.
  • Update the "weekly overview" table of this issue to also include:
    • any blocking CRs
  • Update the Deployments metric review epic.
    • Add a new row to the Overview section: Copy and paste the information from the Overview section in this issue and link to this issue in the Breakdown of blockers column.
    • Update the Graph: Update the data on the spreadsheet and then update the graph on the Deployments metric review epic.

📣 The deployment blockers per week can now be visualized in the Deployment Blockers Dashboard.

Edited by Maina Ng'ang'a