Deployment Blockers - Week: 2026-05-11-2026-05-17

Overview

Start date End date Production deployments blocked for # of blockers
2026-05-11 2026-05-17 62.0 61

Blocker types overview

Blocker Type Time blocked on gstg Time blocked on gprd
RootCauseFlaky-Test 5.5 5.5
RootCauseIndeterminate 0.5 0.5
RootCauseSoftware-Change 41.0 41.0
Change-request 0.0 1.0
RootCauseSaturation 6.0 6.0
RootCauseKnown-Software-Issue 8.0 8.0
Total 61.0 62.0

Weekly overview

Issue Blocker type gstg gprd
2026-05-15: Mattermost removal from Omnibus bre... (gitlab-com/gl-infra/production#22132 - closed) RootCauseSoftware-Change 14.0 14.0
2026-05-15: Loadbalancer error rate for web ser... (gitlab-com/gl-infra/production#22127 - closed) RootCauseSoftware-Change 10.0 10.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22049+ RootCauseSoftware-Change 7.0 7.0
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22052+ RootCauseSoftware-Change 7.0 7.0
https://gitlab.com/gitlab-org/release/tasks/-/work_items/26146+ RootCauseKnown-Software-Issue 6.0 6.0
2026-05-14: QA smoke jobs failing due to stagin... (gitlab-com/gl-infra/production#22118 - closed) RootCauseSoftware-Change 3.0 3.0
2026-05-12: Error rates violating SLO (gitlab-com/gl-infra/production#22103 - closed) RootCauseSaturation 3.0 3.0
2026-05-12: Sidekiq queueing apdex SLO violatio... (gitlab-com/gl-infra/production#22057 - closed) RootCauseSaturation 3.0 3.0
Failing QA jobs due to expired token `GITLAB_QA... (#26148 - closed) RootCauseKnown-Software-Issue 2.0 2.0
2026-05-13: rename Ops GitLab Runner Helm release (gitlab-com/gl-infra/production#22050 - closed) Change-request 0.0 1.0
QA failure: qa/specs/features/browser_ui/2_plan... (#26220 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/api/10_software_s... (#26095 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/api/10_software_s... (#26213 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/api/10_software_s... (#26194 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: (#26164 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/ee/api/2_plan/epi... (#26114 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/6_depl... (#26116 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/ee/browser_ui/18_... (#26074 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/api/10_software_s... (#26027 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#26035 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#26054 - closed) RootCauseFlaky-Test 0.5 0.5
2026-05-11: 500 errors when trying to access it... (gitlab-com/gl-infra/production#22041 - closed) RootCauseIndeterminate 0.5 0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/26119+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26121+ 0.0 0.0
QA failures on gstg-cny (#26140 - closed) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26142+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26254+ 0.0 0.0
QA failure: qa/specs/features/ee/browser_ui/18_... (#26153 - closed) 0.0 0.0
2026-05-15: Code Review and custom flows faillu... (gitlab-com/gl-infra/production#22135 - closed) 0.0 0.0
QA failure: qa/specs/features/ee/browser_ui/18_... (#26158 - closed) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26162+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26163+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26165+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26172+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26177+ 0.0 0.0
QA failures on gstg-cny (#26227 - closed) 0.0 0.0
`gstg-cny-gitaly` deployment failures due to st... (#26224 - closed) 0.0 0.0
2026-05-11: Redis memory utilization on shared-... (gitlab-com/gl-infra/production#22045 - closed) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26050+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26056+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26085+ 0.0 0.0
2026-05-12: Redis primary CPU utilization for r... (gitlab-com/gl-infra/production#22060 - closed) 0.0 0.0
QA failures on gprd-cny (#26089 - closed) 0.0 0.0
2026-05-12: Loadbalancer error rate for web ser... (gitlab-com/gl-infra/production#22107 - closed) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26096+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26097+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26098+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26099+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26100+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26101+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26102+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26103+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26104+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26105+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26106+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26107+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26108+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26109+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26110+ 0.0 0.0
2026-05-13: Sidekiq queueing SLO violation on u... (gitlab-com/gl-infra/production#22115 - closed) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/26118+ 0.0 0.0
Total 61.0 62.0

Additional incidents

Below is a list of production incidents created last week.

Click to expand
Issue
2026-05-17: Gitaly goserver apdex SLO violation... (gitlab-com/gl-infra/production#22141 - closed)
2026-05-17: Prometheus unable to scrape exporte... (gitlab-com/gl-infra/production#22140 - closed)
2026-05-17: gitlab-restore postgres-prdsub snit... (gitlab-com/gl-infra/production#22139 - closed)
2026-05-16: Sidekiq execution SLI below apdex S... (gitlab-com/gl-infra/production#22138 - closed)
2026-05-15: Cells in Staging Have Missing DNS (gitlab-com/gl-infra/production#22133 - closed)
2026-05-15: Cancelling flow sessions returning ... (gitlab-com/gl-infra/production#22131 - closed)
2026-05-15: The Disk inode Utilization per Devi... (gitlab-com/gl-infra/production#22130 - closed)
2026-05-15: No traffic reported on pubsub_topic... (gitlab-com/gl-infra/production#22129 - closed)
2026-05-15: No traffic reported by mimir_query_... (gitlab-com/gl-infra/production#22128 - closed)
2026-05-15: gstg-cny-gitaly job failures due to... (gitlab-com/gl-infra/production#22126 - closed)
2026-05-14: Grafana requests to mimir-gitlab-op... (gitlab-com/gl-infra/production#22124 - closed)
2026-05-14: Disk space for postgres-archive mai... (gitlab-com/gl-infra/production#22123 - closed)
2026-05-14: Custom flows not working on GitLab.com (gitlab-com/gl-infra/production#22120 - closed)
2026-05-13: Error rate for Grafana requests to ... (gitlab-com/gl-infra/production#22116 - closed)
https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22114+
2026-05-13: Some pipelines fail to run dependin... (gitlab-com/gl-infra/production#22113 - closed)
2026-05-13: CNG gitaly Job unable to clone Secu... (gitlab-com/gl-infra/production#22111 - closed)
2026-05-12: At least 10% of chef-runs are faili... (gitlab-com/gl-infra/production#22106 - closed)
2026-05-12: AI Gateway deployments failing (gitlab-com/gl-infra/production#22105 - closed)
2026-05-12: External DNS records are stale (gitlab-com/gl-infra/production#22058 - closed)
2026-05-12: 401 errors when using git ssh (gitlab-com/gl-infra/production#22054)
2026-05-12: Some customers reporting 401 errors... (gitlab-com/gl-infra/production#22053 - closed)
2026-05-12: Increase of errors for Hosted runne... (gitlab-com/gl-infra/production#22051 - closed)
2026-05-11: The ci_runner_jobs SLI of the ci-ru... (gitlab-com/gl-infra/production#22046)
2026-05-11: Loadbalancer error rate SLO violati... (gitlab-com/gl-infra/production#22044 - closed)
2026-05-11: External DNS records are stale in g... (gitlab-com/gl-infra/production#22042 - closed)

Instructions

  • Review the Additional incidents list and add the Deploys-blocked and RootCause labels if required.
  • Retrigger this pipeline job to update this report and metrics with the updated values. Consider this will discard any manual changes added.
  • Update the "weekly overview" table of this issue to also include:
    • any blocking CRs
  • Update the Deployments metric review epic.
    • Add a new row to the Overview section: Copy and paste the information from the Overview section in this issue and link to this issue in the Breakdown of blockers column.
    • Update the Graph: Update the data on the spreadsheet and then update the graph on the Deployments metric review epic.

📣 The deployment blockers per week can now be visualized in the Deployment Blockers Dashboard.

Edited by GitLab Release Tools Bot