Deployment Blockers - Week: 2026-02-02-2026-02-08

Overview

Start date End date Production deployments blocked for # of blockers
2026-02-02 2026-02-08 36.0 28

Blocker types overview

Blocker Type Time blocked on gstg Time blocked on gprd
RootCauseSoftware-Change 24.0 26.0
RootCauseFlaky-Test 5.0 5.0
RootCauseConfig-Change 5.0 5.0
Total 34.0 36.0

Weekly overview

Issue Blocker type gstg gprd
2026-02-02: Links on MRs and User dashboard iss... (gitlab-com/gl-infra/production#21183 - closed) RootCauseSoftware-Change 9.0 9.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/23315+ 0.0 0.0
QA failure: qa/specs/features/browser_ui/8_moni... (#23330 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#23334 - closed) RootCauseSoftware-Change 12.0 12.0
2026-02-03: Sidekiq queueing SLI below SLO on l... (gitlab-com/gl-infra/production#21190 - closed) RootCauseSoftware-Change 3.0 5.0
2026-02-03: Redis deployment error causes delay... (gitlab-com/gl-infra/production#21192 - closed) RootCauseConfig-Change 5.0 5.0
QA failures on gprd-cny (#23369 - closed) 0.0 0.0
QA failures on gprd-cny (#23370 - closed) 0.0 0.0
QA failures on gprd-cny (#23371 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gprd-cny (#23373 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gprd-cny (#23377 - closed) 0.0 0.0
QA failure: qa/specs/features/browser_ui/8_moni... (#23380 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gprd-cny (#23383 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/23384+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/23386+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/23388+ 0.0 0.0
QA failure: qa/specs/features/browser_ui/2_plan... (#23398 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#23427 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#23429 - closed) 0.0 0.0
QA failure: (#23433 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/23442+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/23453+ 0.0 0.0
QA failure: qa/specs/features/browser_ui/3_crea... (#23456 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/23462+ 0.0 0.0
QA failure: qa/specs/features/ee/browser_ui/2_p... (#23470 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/23472+ 0.0 0.0
2026-02-06: Workhorse error rate violating SLO ... (gitlab-com/gl-infra/production#21226 - closed) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/23473+ 0.0 0.0
Total 34.0 36.0

Additional incidents

Below is a list of production incidents created last week.

Click to expand
Issue
2026-02-08: Apdex SLO violation for inference_a... (gitlab-com/gl-infra/production#21233 - closed)
2026-02-08: Apdex SLO violation for anthropic m... (gitlab-com/gl-infra/production#21232 - closed)
2026-02-08: Patroni GCS snapshot did not comple... (gitlab-com/gl-infra/production#21231 - closed)
2026-02-08: Loadbalancer error rate for package... (gitlab-com/gl-infra/production#21230 - closed)
2026-02-06: Inference error rate for anthropic ... (gitlab-com/gl-infra/production#21229 - closed)
2026-02-06: Sidekiq queueing apdex SLO violatio... (gitlab-com/gl-infra/production#21228 - closed)
2026-02-06: E2E Test Failures Blocking 18.8 Sec... (gitlab-com/gl-infra/production#21227 - closed)
2026-02-06: Elevated error rate on runway_ingre... (gitlab-com/gl-infra/production#21222 - closed)
2026-02-05: Loadbalancer error rate for web ser... (gitlab-com/gl-infra/production#21220 - closed)
2026-02-05: Sidekiq queueing apdex SLO violatio... (gitlab-com/gl-infra/production#21218 - closed)
2026-02-05: Sidekiq jobs on low-urgency-cpu-bou... (gitlab-com/gl-infra/production#21216 - closed)
2026-02-05: The polling SLI of the ci-runners s... (gitlab-com/gl-infra/production#21213 - closed)
2026-02-04: The loadbalancer SLI of the ci-runn... (gitlab-com/gl-infra/production#21212 - closed)
2026-02-04: Apdex SLO violation for inference_v... (gitlab-com/gl-infra/production#21209 - closed)
2026-02-04: post deploy migration failing on pr... (gitlab-com/gl-infra/production#21206 - closed)
2026-02-04: The polling SLI of the ci-runners s... (gitlab-com/gl-infra/production#21205 - closed)
2026-02-04: No traffic received by pulp_app_tas... (gitlab-com/gl-infra/production#21204 - closed)
2026-02-03: The queuing_queries_duration SLI of... (gitlab-com/gl-infra/production#21203 - closed)
2026-02-03: Long-running transaction on patroni... (gitlab-com/gl-infra/production#21202 - closed)
2026-02-03: Apdex SLO violation for inference_v... (gitlab-com/gl-infra/production#21199 - closed)
2026-02-03: GraphQL query apdex SLO violation i... (gitlab-com/gl-infra/production#21197 - closed)
2026-02-03: Error rate for anthropic inferences... (gitlab-com/gl-infra/production#21196 - closed)
2026-02-03: Apdex SLO violation for GraphQL que... (gitlab-com/gl-infra/production#21195 - closed)
2026-02-03: Blackbox probes failing for https:/... (gitlab-com/gl-infra/production#21191 - closed)
2026-02-03: Foundational Flows do not cascade c... (gitlab-com/gl-infra/production#21189 - closed)
2026-02-03: Prometheus scrape failures on patro... (gitlab-com/gl-infra/production#21187 - closed)
2026-02-02: Nonprod Elastic cluster unhealthy. (gitlab-com/gl-infra/production#21186 - closed)
2026-02-02: High error rate for anthropic model... (gitlab-com/gl-infra/production#21185 - closed)
2026-02-02: Inline comments on MRs cannot be hi... (gitlab-com/gl-infra/production#21184 - closed)
2026-02-02: Gitaly goserver error rate SLO viol... (gitlab-com/gl-infra/production#21182)
2026-02-02: Long-running transaction over 10m b... (gitlab-com/gl-infra/production#21181 - closed)
2026-02-02: Scheduled gitlab-restore job for po... (gitlab-com/gl-infra/production#21180 - closed)

Instructions

  • Review the Additional incidents list and add the Deploys-blocked and RootCause labels if required.
  • Retrigger this pipeline job to update this report and metrics with the updated values. Consider this will discard any manual changes added.
  • Update the "weekly overview" table of this issue to also include:
    • any blocking CRs
  • Update the Deployments metric review epic.
    • Add a new row to the Overview section: Copy and paste the information from the Overview section in this issue and link to this issue in the Breakdown of blockers column.
    • Update the Graph: Update the data on the spreadsheet and then update the graph on the Deployments metric review epic.

📣 The deployment blockers per week can now be visualized in the Deployment Blockers Dashboard.

Edited by Maina Ng'ang'a