Deployment Blockers - Week: 2026-06-29-2026-07-05

Overview

Start date End date Production deployments blocked for # of blockers
2026-06-29 2026-07-05 60.5 37

Blocker types overview

Blocker Type Time blocked on gstg Time blocked on gprd
RootCauseFlaky-Test 11.5 11.5
RootCauseSaturation 4.0 4.0
RootCauseMalicious-Traffic 3.0 3.0
RootCauseSoftware-Change 3.0 3.0
Total 21.5 21.5

Weekly overview

Issue Blocker type gstg gprd
[C2] [GPRD] [2026-06-27 to 2026-06-30] - Upgrad... (gitlab-com/gl-infra/production#22363 - closed) 39.0 39.0
2026-06-30: CI/CD pipelines not being triggered... (gitlab-com/gl-infra/production#22412 - closed) RootCauseSaturation 3.0 3.0
2026-06-30: Gitaly goserver apdex SLO violation... (gitlab-com/gl-infra/production#22418 - closed) RootCauseMalicious-Traffic 3.0 3.0
2026-06-30: server_code_completions error rate ... (gitlab-com/gl-infra/production#22424 - closed) RootCauseSoftware-Change 3.0 3.0
2026-07-01: CI pipelines stuck in pending state. (gitlab-com/gl-infra/production#22433) RootCauseSaturation 1.0 1.0
QA failure: qa/specs/features/browser_ui/4_veri... (#27397 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#27401 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/10_sof... (#27405 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/4_veri... (#27411 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gprd-cny (#27417 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/ee/browser_ui/2_p... (#27418 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: (#27419 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#27420 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: (#27421 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#27422 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/ee/api/2_plan/epi... (#27423 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: (#27425 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: (#27427 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/ee/api/2_plan/epi... (#27390 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/10_sof... (#27379 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gprd-cny (#27370 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gprd-cny (#27369 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/ee/api/2_plan/epi... (#27367 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/ee/browser_ui/2_p... (#27354 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/ee/api/2_plan/epi... (#27351 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gprd-cny (#27350 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: (#27344 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/10_sof... (#27336 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/27430+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27429+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27428+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27426+ 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22409+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27424+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27371+ 0.0 0.0
QA failures on gstg-cny (#27391 - closed) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/27394+ 0.0 0.0
Total 21.5 21.5

Additional incidents

Below is a list of production incidents created last week.

Click to expand
Issue
2026-07-05: workhorse_auth_api error rate viola... (gitlab-com/gl-infra/production#22455 - closed)
2026-07-05: workhorse_auth_api error rate 0.227... (gitlab-com/gl-infra/production#22454 - closed)
2026-07-04: server_code_completions apdex dip i... (gitlab-com/gl-infra/production#22453 - closed)
2026-07-03: False alarms for Gitaly traffic sto... (gitlab-com/gl-infra/production#22451 - closed)
2026-07-03: Bedrock models not working in DAP (gitlab-com/gl-infra/production#22449 - closed)
2026-07-03: GCP scheduled snapshots delayed for... (gitlab-com/gl-infra/production#22448 - closed)
2026-07-03: The runner_api_requests SLI of the ... (gitlab-com/gl-infra/production#22447 - closed)
2026-07-02: The ci_runner_jobs SLI of the ci-ru... (gitlab-com/gl-infra/production#22446 - closed)
2026-07-02: Mimir query frontend, scheduler, an... (gitlab-com/gl-infra/production#22443 - closed)
2026-07-02: CloudSQL CustomersDot CPU utilizati... (gitlab-com/gl-infra/production#22441 - closed)
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22439+
2026-07-01: Sidekiq queueing apdex below SLO on... (gitlab-com/gl-infra/production#22436 - closed)
2026-07-01: Multiple gitlab pods Unschedulable ... (gitlab-com/gl-infra/production#22435 - closed)
2026-07-01: Sidekiq queueing apdex at 79.76% on... (gitlab-com/gl-infra/production#22432 - closed)
2026-07-01: server_foundational error rate at 5... (gitlab-com/gl-infra/production#22429 - closed)
2026-07-01: runway_ingress error rate at 15.53%... (gitlab-com/gl-infra/production#22428 - closed)
2026-06-30: gitaly goserver traffic ceased on g... (gitlab-com/gl-infra/production#22425 - closed)
2026-06-30: Workhorse error rate violating SLO ... (gitlab-com/gl-infra/production#22422 - closed)
2026-06-30: Blackbox probe availability at 70.5... (gitlab-com/gl-infra/production#22421 - closed)
2026-06-30: Patroni is down on patroni-sec-v17 (gitlab-com/gl-infra/production#22416 - closed)
2026-06-30: Sidekiq queueing apdex SLO violatio... (gitlab-com/gl-infra/production#22415 - closed)
2026-06-30: rails_primary_sql apdex 97.89% in p... (gitlab-com/gl-infra/production#22411 - closed)
2026-06-30: CloudSQL CustomersDot CPU utilizati... (gitlab-com/gl-infra/production#22410 - closed)
2026-06-29: Gitaly goserver error rate at 1.147... (gitlab-com/gl-infra/production#22407 - closed)
2026-06-29: Blackbox probe availability below 6... (gitlab-com/gl-infra/production#22406 - closed)
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22404+
2026-06-29: CloudSQL CustomersDot CPU utilizati... (gitlab-com/gl-infra/production#22401 - closed)
2026-06-29: No Gitaly goserver traffic on gital... (gitlab-com/gl-infra/production#22400 - closed)

Instructions

  • Review the Additional incidents list and add the Deploys-blocked and RootCause labels if required.
  • Retrigger this pipeline job to update this report and metrics with the updated values. Consider this will discard any manual changes added.
  • Update the "weekly overview" table of this issue to also include:
    • any blocking CRs
  • Update the Deployments metric review epic.
    • Add a new row to the Overview section: Copy and paste the information from the Overview section in this issue and link to this issue in the Breakdown of blockers column.
    • Update the Graph: Update the data on the spreadsheet and then update the graph on the Deployments metric review epic.

📣 The deployment blockers per week can now be visualized in the Deployment Blockers Dashboard.

Edited by Dat Tang