Deployment Blockers - Week: 2026-03-09-2026-03-15

Overview

Start date End date Production deployments blocked for # of blockers
2026-03-09 2026-03-15 43.0 43

Blocker types overview

Blocker Type Time blocked on gstg Time blocked on gprd
RootCauseSaturation 11.5 11.5
RootCauseFlaky-Test 7.5 7.5
RootCauseIndeterminate 0.5 0.5
RootCauseSoftware-Change 17.0 17.0
RootCauseMalicious-Traffic 0.5 0.5
RootCauseDB-Migration 1.0 1.0
RootCauseKnown-Software-Issue 3.0 3.0
RootCauseConfig-Change 2.0 2.0
Total 43.0 43.0

Weekly overview

Issue Blocker type gstg gprd
2026-03-09: Sidekiq queueing SLO violation on c... (gitlab-com/gl-infra/production#21484 - closed) RootCauseSaturation 3.0 3.0
2026-03-09: Customers reporting delays in pipel... (gitlab-com/gl-infra/production#21485 - closed) 0.0 0.0
2026-03-09: Gitaly goserver apdex SLO violation... (gitlab-com/gl-infra/production#21487 - closed) RootCauseSaturation 2.0 2.0
QA failure: qa/specs/features/browser_ui/9_tena... (#24046 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/24047+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/24049+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/24050+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/24051+ 0.0 0.0
2026-03-10: Sidekiq queueing apdex SLO violatio... (gitlab-com/gl-infra/production#21498 - closed) RootCauseSaturation 5.0 5.0
2026-03-10: KAS pods stuck when terminating (gitlab-com/gl-infra/production#21500 - closed) RootCauseIndeterminate 0.5 0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/24052+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/24053+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/24054+ 0.0 0.0
QA failures on gprd-cny (#24055 - closed) 0.0 0.0
QA failures on gprd-cny (#24056 - closed) 0.0 0.0
2026-03-10: Auto-Deploy broken (gitlab-com/gl-infra/production#21507 - closed) RootCauseSoftware-Change 11.0 11.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/24063+ 0.0 0.0
QA failures on gprd-cny (#24064 - closed) 0.0 0.0
2026-03-11: Open file descriptor utilization in... (gitlab-com/gl-infra/production#21515 - closed) 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21518+ RootCauseMalicious-Traffic 0.5 0.5
QA failure: (#24065 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21519+ RootCauseSoftware-Change 4.0 4.0
QA failure: qa/specs/features/browser_ui/10_sof... (#24066 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/24074+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/24077+ 0.0 0.0
QA failure: (#24080 - closed) RootCauseFlaky-Test 0.5 0.5
2026-03-12: Post-deploy migrations gprd failed ... (gitlab-com/gl-infra/production#21524 - closed) RootCauseDB-Migration 1.0 1.0
QA failures on gprd-cny (#24087 - closed) 0.0 0.0
2026-03-12: Sidekiq jobs queueing duration SLO ... (gitlab-com/gl-infra/production#21528 - closed) RootCauseSaturation 0.5 0.5
QA failures on gprd-cny (#24092 - closed) RootCauseFlaky-Test 4.0 4.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/24107+ 0.0 0.0
2026-03-12: Canary: Custom agents chat not working (gitlab-com/gl-infra/production#21535 - closed) RootCauseSoftware-Change 2.0 2.0
2026-03-12: CNG builds failing on cfssl-self-sign (gitlab-com/gl-infra/production#21537 - closed) RootCauseKnown-Software-Issue 3.0 3.0
2026-03-13: No worker_requests traffic reported... (gitlab-com/gl-infra/production#21540 - closed) 0.0 0.0
QA failure: (#24128 - closed) RootCauseFlaky-Test 0.5 0.5
2026-03-13: gitlab-zoekt in Staging is stuck (gitlab-com/gl-infra/production#21542 - closed) RootCauseConfig-Change 2.0 2.0
QA failure: (#24130 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: (#24132 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gprd-cny (#24133 - closed) RootCauseSaturation 1.0 1.0
QA failures on gprd-cny (#24134 - closed) 0.0 0.0
QA failure: qa/specs/features/api/4_verify/file... (#24137 - closed) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/24139+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/24140+ 0.0 0.0
Total 43.0 43.0

Additional incidents

Below is a list of production incidents created last week.

Click to expand
Issue
2026-03-14: GraphQL query error rate exceeding ... (gitlab-com/gl-infra/production#21555 - closed)
2026-03-14: GraphQL query error rate violating ... (gitlab-com/gl-infra/production#21554 - closed)
2026-03-14: runway_ingress apdex SLO violation ... (gitlab-com/gl-infra/production#21553 - closed)
2026-03-13: Sidekiq queueing SLI apdex SLO viol... (gitlab-com/gl-infra/production#21552 - closed)
2026-03-13: Helm upgrade stuck (gitlab-com/gl-infra/production#21551 - closed)
2026-03-13: Availability of compensation calcul... (gitlab-com/gl-infra/production#21550 - closed)
2026-03-13: Sidekiq queueing duration SLO viola... (gitlab-com/gl-infra/production#21549 - closed)
2026-03-13: Production QA failure (gitlab-com/gl-infra/production#21548 - closed)
2026-03-13: Sidekiq job queueing apdex SLO viol... (gitlab-com/gl-infra/production#21547 - closed)
2026-03-13: SLO violation: apdex for Rails prim... (gitlab-com/gl-infra/production#21545 - closed)
2026-03-13: Chef failures on patroni VMs (gitlab-com/gl-infra/production#21543 - closed)
2026-03-13: Pipeline Job Failures: fatal: could... (gitlab-com/gl-infra/production#21539 - closed)
2026-03-12: Sidekiq queueing SLI apdex violatio... (gitlab-com/gl-infra/production#21538 - closed)
2026-03-12: Sidekiq queueing SLI apdex SLO viol... (gitlab-com/gl-infra/production#21536 - closed)
2026-03-12: Mimir/grafana down (gitlab-com/gl-infra/production#21533 - closed)
2026-03-12: Reports of CI Delays in picking up ... (gitlab-com/gl-infra/production#21532 - closed)
2026-03-12: Sidekiq queueing SLO violation on l... (gitlab-com/gl-infra/production#21531 - closed)
2026-03-12: Code review flows fail with NotImpl... (gitlab-com/gl-infra/production#21530 - closed)
2026-03-12: GPT models not working in agentic chat (gitlab-com/gl-infra/production#21529)
2026-03-12: Sidekiq queueing SLO violation on u... (gitlab-com/gl-infra/production#21527 - closed)
2026-03-12: Duo Agents cannot be selected with ... (gitlab-com/gl-infra/production#21526 - closed)
2026-03-12: No worker_requests traffic reported... (gitlab-com/gl-infra/production#21523 - closed)
2026-03-11: Loadbalancer error rate for package... (gitlab-com/gl-infra/production#21522 - closed)
2026-03-11: Omnibus package check failed for so... (gitlab-com/gl-infra/production#21517 - closed)
2026-03-11: Sidekiq queueing apdex SLO violatio... (gitlab-com/gl-infra/production#21513 - closed)
2026-03-11: Persistent Volume Claim inode utili... (gitlab-com/gl-infra/production#21512 - closed)
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21506+
2026-03-10: Error rate for runway_ingress on ex... (gitlab-com/gl-infra/production#21505 - closed)
2026-03-10: External DNS records have not been ... (gitlab-com/gl-infra/production#21504 - closed)
2026-03-10: us-east1-c traffic absent (gitlab-com/gl-infra/production#21503 - closed)
2026-03-10: Apdex SLO violation for rails_prima... (gitlab-com/gl-infra/production#21502 - closed)
2026-03-10: Staging: Zoekt webserver search not... (gitlab-com/gl-infra/production#21501 - closed)
2026-03-10: Users reporting 403 Forbidden respo... (gitlab-com/gl-infra/production#21499 - closed)
2026-03-10: Sidekiq queueing apdex SLO violatio... (gitlab-com/gl-infra/production#21495 - closed)
2026-03-09: 2026-03-09 | Daily Data Team Sec Re... (gitlab-com/gl-infra/production#21492 - closed)
2026-03-09: Sidekiq queueing apdex SLO violatio... (gitlab-com/gl-infra/production#21490 - closed)
2026-03-09: SQL transaction apdex SLO violation... (gitlab-com/gl-infra/production#21483 - closed)
2026-03-09: Sidekiq queueing SLI apdex SLO viol... (gitlab-com/gl-infra/production#21482 - closed)
2026-03-09: The polling SLI of the ci-runners s... (gitlab-com/gl-infra/production#21481 - closed)
2026-03-09: The polling SLI of the ci-runners s... (gitlab-com/gl-infra/production#21478 - closed)

Instructions

  • Review the Additional incidents list and add the Deploys-blocked and RootCause labels if required.
  • Retrigger this pipeline job to update this report and metrics with the updated values. Consider this will discard any manual changes added.
  • Update the "weekly overview" table of this issue to also include:
    • any blocking CRs
  • Update the Deployments metric review epic.
    • Add a new row to the Overview section: Copy and paste the information from the Overview section in this issue and link to this issue in the Breakdown of blockers column.
    • Update the Graph: Update the data on the spreadsheet and then update the graph on the Deployments metric review epic.

📣 The deployment blockers per week can now be visualized in the Deployment Blockers Dashboard.

Edited by GitLab Release Tools Bot