Deployment Blockers - Week: 2026-10-12-2026-10-18

Overview

Start date End date Production deployments blocked for # of blockers
2026-10-12 2026-10-18 62.5 97

Blocker types overview

Blocker Type Time blocked on gstg Time blocked on gprd
RootCauseFlaky-Test 2.5 2.5
RootCauseSoftware-Change 18.0 18.0
Total 20.5 20.5

Weekly overview

Issue Blocker type gstg gprd
2026-10-12: Registry pods crashing in gprd due ... (gitlab-com/gl-infra/production#23166 - closed) RootCauseSoftware-Change 14.0 14.0
2026-10-14: gstg-cny deployments crash on remov... (gitlab-com/gl-infra/production#23188 - closed) 8.0 8.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23192+ 7.0 7.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23170+ 6.0 6.0
2026-10-15: Blackbox probe availability for cus... (gitlab-com/gl-infra/production#23240 - closed) 5.0 5.0
2026-10-17: auto-deploy blocked due to bad migr... (gitlab-com/gl-infra/production#23268 - closed) RootCauseSoftware-Change 4.0 4.0
2026-10-13: ai-assisted webservice fails deploy... (gitlab-com/gl-infra/production#23171 - closed) 4.0 4.0
2026-10-12: tool_use error rate at 30.25% in du... (gitlab-com/gl-infra/production#23158 - closed) 3.0 3.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23261+ 2.0 2.0
2026-10-13: PGBouncer Client Connections per Pr... (gitlab-com/gl-infra/production#23175 - closed) 2.0 2.0
2026-10-15: HTTP 502 on gitlab.com/users/sign_in (gitlab-com/gl-infra/production#23239 - closed) 1.0 1.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23250+ 1.0 1.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23251+ 1.0 1.0
2026-10-16: GLQL returning stale data (gitlab-com/gl-infra/production#23253 - closed) 1.0 1.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23186+ 1.0 1.0
QA failures on gprd-cny (#28051 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gprd-cny (#28055 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/api/3_create/merg... (#28087 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gprd-cny (#28044 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: (#28070 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/28069+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28067+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28066+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28065+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28064+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28063+ 0.0 0.0
QA failures on gprd-cny (#28060) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28059+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28058+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28057+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28056+ 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23233+ 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23232+ 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23231+ 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23230+ 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23229+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28088+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28086+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28085+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28084+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28083+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28082+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28081+ 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23262+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28080+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28079+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28078+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28077+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28075+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28074+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28073+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28072+ 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23257+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28071+ 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23228+ 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23208+ 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23207+ 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23206+ 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23205+ 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23204+ 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23203+ 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23201+ 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23200+ 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23199+ 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23198+ 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23197+ 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23196+ 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23195+ 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23194+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28050+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28049+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28048+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28047+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28046+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/28045+ 0.0 0.0
Enable Topology Service client for gitlab-shell... (gitlab-com/gl-infra/production#22944) Change-request 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23227+ 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23226+ 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23225+ 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23224+ 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23223+ 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23222+ 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23221+ 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23220+ 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23219+ 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23218+ 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23217+ 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23216+ 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23215+ 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23214+ 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23213+ 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23212+ 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23211+ 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23210+ 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23209+ 0.0 0.0
QA failures on gprd-cny (#28054) 0.0 0.0
QA failures on gprd-cny (#28053) 0.0 0.0
Total 62.5 62.5

Additional incidents

Below is a list of production incidents created last week.

Click to expand
Issue
2026-10-18: Unable to view Changes tab for any ... (gitlab-com/gl-infra/production#23279 - closed)
2026-10-18: Unable to load commit history for f... (gitlab-com/gl-infra/production#23278)
2026-10-18: Gitaly goserver apdex SLO violation... (gitlab-com/gl-infra/production#23270 - closed)
2026-10-17: web loadbalancer and workhorse erro... (gitlab-com/gl-infra/production#23269 - closed)
2026-10-16: Loadbalancer SLI error rate 1.976% ... (gitlab-com/gl-infra/production#23267 - closed)
2026-10-16: Missing logs for gprd main (gitlab-com/gl-infra/production#23266 - closed)
2026-10-16: Siphon logical replication lag on p... (gitlab-com/gl-infra/production#23256 - closed)
2026-10-16: Redis max memory utilization nearin... (gitlab-com/gl-infra/production#23255 - closed)
2026-10-15: PGBouncer Primary_Main client conne... (gitlab-com/gl-infra/production#23244 - closed)
2026-10-15: server_unknown error rate at 30.41%... (gitlab-com/gl-infra/production#23242 - closed)
2026-10-14: runway_ingress SLI error rate at 30... (gitlab-com/gl-infra/production#23238 - closed)
2026-10-14: qa tests failing after several retries (gitlab-com/gl-infra/production#23235 - closed)
2026-10-14: The ci_runner_jobs SLI of the ci-ru... (gitlab-com/gl-infra/production#23234 - closed)
2026-10-14: PGBouncer Client Connections per Pr... (gitlab-com/gl-infra/production#23190)
2026-10-14: Compute Engine disk patroni-main-v1... (gitlab-com/gl-infra/production#23189 - closed)
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23185+
2026-10-13: Blackbox probe availability for htt... (gitlab-com/gl-infra/production#23182 - closed)
2026-10-13: Web loadbalancer and workhorse erro... (gitlab-com/gl-infra/production#23180 - closed)
2026-10-13: Sidekiq queueing apdex at 86.06% on... (gitlab-com/gl-infra/production#23179 - closed)
2026-10-13: pre deploy failed due to gitlab-git... (gitlab-com/gl-infra/production#23178 - closed)
2026-10-13: PubSub queuing high on pubsub-sidek... (gitlab-com/gl-infra/production#23177 - closed)
2026-10-13: amd64-small-linux GCP 503s causing ... (gitlab-com/gl-infra/production#23176 - closed)
2026-10-13: Sidekiq queueing apdex at 96.25% on... (gitlab-com/gl-infra/production#23174 - closed)
2026-10-13: The ci_runner_jobs SLI of the ci-ru... (gitlab-com/gl-infra/production#23173 - closed)
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23169+
2026-10-13: Web service loadbalancer error rate... (gitlab-com/gl-infra/production#23167 - closed)
2026-10-12: pubsub_subscriptions traffic absent... (gitlab-com/gl-infra/production#23163 - closed)
2026-10-12: Siphon's PostgreSQL logical replica... (gitlab-com/gl-infra/production#23161 - closed)
2026-10-12: gitlab-restore/postgres-gprd missing (gitlab-com/gl-infra/production#23152 - closed)

Instructions

  • Review the Additional incidents list and add the Deploys-blocked and RootCause labels if required.
  • Retrigger this pipeline job to update this report and metrics with the updated values. Consider this will discard any manual changes added.
  • Update the "weekly overview" table of this issue to also include:
    • any blocking CRs
  • Update the Deployments metric review epic.
    • Add a new row to the Overview section: Copy and paste the information from the Overview section in this issue and link to this issue in the Breakdown of blockers column.
    • Update the Graph: Update the data on the spreadsheet and then update the graph on the Deployments metric review epic.

📣 The deployment blockers per week can now be visualized in the Deployment Blockers Dashboard.

Edited by Manjunath Bhadrannavar