Deployment Blockers - Week: 2026-03-02-2026-03-08

Overview

Start date End date Production deployments blocked for # of blockers
2026-03-02 2026-03-08 21.5 39

Blocker types overview

Blocker Type Time blocked on gstg Time blocked on gprd
Change-request 4.0 4.0
RootCauseFlaky-Test 6.5 6.5
RootCauseExternal-Dependency 3.0 3.0
RootCauseMalicious-Traffic 1.0 1.0
RootCauseDB-Migration 2.0 2.0
RootCauseConfig-Change 2.0 2.0
RootCauseKnown-Software-Issue 3.0 3.0
Total 21.5 21.5

Weekly overview

Issue Blocker type gstg gprd
2026-03-02 : [GSTG] Configure HAProxy to suppor... (gitlab-com/gl-infra/production#21287 - closed) Change-request 4.0 4.0
2026-03-05: Enable workload identity and Signed... (gitlab-com/gl-infra/production#21392 - closed) Change-request 0.0 0.0
2026-03-02: Zoekt code search result fails (gitlab-com/gl-infra/production#21421 - closed) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/23921+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/23923+ 0.0 0.0
QA failure: (#23934 - closed) RootCauseFlaky-Test 0.5 0.5
2026-03-03: Flows failing with In Memory Error (gitlab-com/gl-infra/production#21429 - closed) 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21433+ 0.0 0.0
QA failure: qa/specs/features/browser_ui/3_crea... (#23940 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#23941 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gprd-cny (#23943 - closed) 0.0 0.0
2026-03-03: Intemrittent 400 errors when loggin... (gitlab-com/gl-infra/production#21436 - closed) RootCauseExternal-Dependency 3.0 3.0
QA failure: qa/specs/features/browser_ui/3_crea... (#23945 - closed) 0.0 0.0
QA failure: qa/specs/features/browser_ui/10_sof... (#23948 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#23957 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#23961 - closed) RootCauseFlaky-Test 0.5 0.5
2026-03-04: HTTP 503 server unavailable in gitl... (gitlab-com/gl-infra/production#21439 - closed) 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21440+ RootCauseMalicious-Traffic 1.0 1.0
2026-03-04: ClickHouse Post Deployment Migratio... (gitlab-com/gl-infra/production#21444 - closed) RootCauseDB-Migration 2.0 2.0
QA failure: qa/specs/features/ee/browser_ui/2_p... (#23968 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/23970+ 0.0 0.0
2026-03-04: Prod deployment failed because of n... (gitlab-com/gl-infra/production#21446 - closed) RootCauseConfig-Change 1.0 1.0
QA failures on gstg-cny (#23974 - closed) 0.0 0.0
2026-03-04: gprd-cny deployment failed because ... (gitlab-com/gl-infra/production#21447 - closed) RootCauseConfig-Change 1.0 1.0
QA failures on gstg-cny (#23975 - closed) RootCauseFlaky-Test 0.5 0.5
2026-03-04: Duo agentic chat losing messages wi... (gitlab-com/gl-infra/production#21448 - closed) RootCauseKnown-Software-Issue 1.0 1.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/23978+ 0.0 0.0
QA failure: qa/specs/features/browser_ui/3_crea... (#23979 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/23984+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/23986+ 0.0 0.0
QA failures on gprd-cny (#23989 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/4_veri... (#23990 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/23993+ 0.0 0.0
2026-03-05: CI/CD Pipelines stuck in running st... (gitlab-com/gl-infra/production#21462 - closed) RootCauseKnown-Software-Issue 2.0 2.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21464+ 0.0 0.0
QA failures on gprd-cny (#24017 - closed) 0.0 0.0
QA failures on gstg-cny (#24018 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/24034+ 0.0 0.0
QA failure: qa/specs/features/browser_ui/8_moni... (#24035 - closed) RootCauseFlaky-Test 0.5 0.5
Total 21.5 21.5

Additional incidents

Below is a list of production incidents created last week.

Click to expand
Issue
2026-03-08: Apdex SLO violation for anthropic m... (gitlab-com/gl-infra/production#21476 - closed)
2026-03-08: git pull failures on gitlab main repo (gitlab-com/gl-infra/production#21475 - closed)
2026-03-08: Apdex SLO violation for anthropic m... (gitlab-com/gl-infra/production#21474 - closed)
2026-03-08: Loadbalancer error rate for web ser... (gitlab-com/gl-infra/production#21473 - closed)
2026-03-07: GraphQL query error rate SLO violat... (gitlab-com/gl-infra/production#21472 - closed)
2026-03-06: Sidekiq queueing SLI apdex SLO viol... (gitlab-com/gl-infra/production#21467 - closed)
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21466+
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21457+
2026-03-04: JWT signature verification failures... (gitlab-com/gl-infra/production#21449)
2026-03-04: The loadbalancer SLI of the ci-runn... (gitlab-com/gl-infra/production#21441)
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21435+
2026-03-03: Sidekiq queueing SLI apdex SLO viol... (gitlab-com/gl-infra/production#21428 - closed)
2026-03-02: No traffic reported by pubsub_topic... (gitlab-com/gl-infra/production#21426 - closed)
2026-03-02: Apdex SLO violation for Rails prima... (gitlab-com/gl-infra/production#21423 - closed)
2026-02-28: The runner_api_requests SLI of the ... (gitlab-com/gl-infra/production#21420 - closed)
2026-03-02: The polling SLI of the ci-runners s... (gitlab-com/gl-infra/production#21419 - closed)
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21418+
2026-03-02: Missing heartbeat for gitlab-restor... (gitlab-com/gl-infra/production#21417 - closed)

Instructions

  • Review the Additional incidents list and add the Deploys-blocked and RootCause labels if required.
  • Retrigger this pipeline job to update this report and metrics with the updated values. Consider this will discard any manual changes added.
  • Update the "weekly overview" table of this issue to also include:
    • any blocking CRs
  • Update the Deployments metric review epic.
    • Add a new row to the Overview section: Copy and paste the information from the Overview section in this issue and link to this issue in the Breakdown of blockers column.
    • Update the Graph: Update the data on the spreadsheet and then update the graph on the Deployments metric review epic.

📣 The deployment blockers per week can now be visualized in the Deployment Blockers Dashboard.

Edited by Mawreen Dela Cruz