Deployment Blockers - Week: 2025-05-19-2025-05-25

Overview

Start date End date Production deployments blocked for # of blockers
2025-05-19 2025-05-25 18.5 45

Blocker types overview

Blocker Type Time blocked on gstg Time blocked on gprd
RootCauseFlaky-Test 6.0 6.0
RootCauseSaturation 1.0 1.0
RootCauseExternal-Dependency 4.0 4.0
RootCauseSoftware-Change 7.0 7.0
Change-request 0.5 0.0
RootCauseConfig-Change 0.0 0.5
Total 18.5 18.5

Weekly overview

Issue Blocker type gstg gprd
QA failure: qa/specs/features/ee/browser_ui/10_... (#19503 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: (#19504 - closed) RootCauseFlaky-Test 0.5 0.5
2025-05-19 : Increase in web errors (gitlab-com/gl-infra/production#19849 - closed) RootCauseSaturation 1.0 1.0
QA failure: qa/specs/features/ee/browser_ui/10_... (#19511 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/10_gov... (#19515 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/ee/browser_ui/10_... (#19517 - closed) RootCauseFlaky-Test 0.5 0.5
Monday 2025-05-19 19:53 UTC - `gitlab-org/gitla... (#19519 - closed) 0.0 0.0
QA failure: qa/specs/features/ee/browser_ui/10_... (#19520 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/19521+ 0.0 0.0
QA failure: qa/specs/features/ee/browser_ui/10_... (#19522 - closed) RootCauseFlaky-Test 0.5 0.5
2025-05-19 : Error rate violation in workhorse ... (gitlab-com/gl-infra/production#19853 - closed) RootCauseExternal-Dependency 4.0 4.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/19523+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/19525+ 0.0 0.0
QA failure: qa/specs/features/ee/browser_ui/10_... (#19526 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/19530+ 0.0 0.0
QA failure: (#19533 - closed) 0.0 0.0
2025-05-20 : Notes not rendering when author is... (gitlab-com/gl-infra/production#19854 - closed) RootCauseSoftware-Change 2.0 2.0
2025-05-20 : Increase in 500 errors (gitlab-com/gl-infra/production#19856 - closed) 0.0 0.0
2025-05-20: Increase in 500 errors (gitlab-com/gl-infra/production#19857 - closed) 0.0 0.0
QA failure: qa/specs/features/ee/browser_ui/18_... (#19536 - closed) 0.0 0.0
Tuesday 2025-05-20 13:01 UTC - `gitlab-org/gitl... (#19537 - closed) 0.0 0.0
Tuesday 2025-05-20 13:03 UTC - `gitlab-org/gitl... (#19538 - closed) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/19539+ 0.0 0.0
Tuesday 2025-05-20 15:42 UTC - `gitlab-org/gitl... (#19541 - closed) 0.0 0.0
QA failure: qa/specs/features/ee/api/3_create/c... (#19542 - closed) RootCauseFlaky-Test 0.5 0.5
https://gitlab.com/gitlab-org/release/tasks/-/issues/19543+ 0.0 0.0
Enable `fast_commit` on canary staging, take 2 (gitlab-com/gl-infra/production#19861 - closed) Change-request 0.5 0.0
2025-05-21 : Many "couldn't find remote ref" er... (gitlab-com/gl-infra/production#19863 - closed) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/19553+ 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/19556+ 0.0 0.0
Wednesday 2025-05-21 19:22 UTC - `gitlab-org/gi... (#19557 - closed) 0.0 0.0
2025-05-22 : GitLab CI jobs failing to pull ima... (gitlab-com/gl-infra/production#19866 - closed) 0.0 0.0
[GPRD] Enable `fast_commit` on canary Gitaly (gitlab-com/gl-infra/production#19870 - closed) Change-request 0.0 0.0
Thursday 2025-05-22 15:36 UTC - `gitlab-org/git... (#19569 - closed) 0.0 0.0
https://gitlab.com/gitlab-org/release/tasks/-/issues/19571+ 0.0 0.0
QA failure: qa/specs/features/browser_ui/5_pack... (#19574 - closed) RootCauseFlaky-Test 0.5 0.5
Thursday 2025-05-22 20:52 UTC - `gitlab-org/git... (#19575 - closed) 0.0 0.0
QA failure: qa/specs/features/ee/browser_ui/18_... (#19577 - closed) RootCauseFlaky-Test 0.5 0.5
2025-05-23 : Duo chat not working on web on canary (gitlab-com/gl-infra/production#19874 - closed) RootCauseSoftware-Change 5.0 5.0
QA failure: qa/specs/features/browser_ui/2_plan... (#19583 - closed) RootCauseFlaky-Test 0.5 0.5
2025-05-23 : Production deployment failed due t... (gitlab-com/gl-infra/production#19878 - closed) RootCauseConfig-Change 0.0 0.5
QA failure: qa/specs/features/browser_ui/10_sof... (#19588 - closed) 0.0 0.0
2025-05-24 : High error rate in transactions_pr... (gitlab-com/gl-infra/production#19886 - closed) 0.0 0.0
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19887+ 0.0 0.0
Total 18.5 18.5

Additional incidents

Below is a list of production incidents created last week.

Click to expand
Issue
2025-05-25 : GitLab FOSS site unreachable due t... (gitlab-com/gl-infra/production#19888 - closed)
2025-05-24 : High error rate in transactions_pr... (gitlab-com/gl-infra/production#19885 - closed)
2025-05-24: Chef client has been disabled for a... (gitlab-com/gl-infra/production#19884 - closed)
2025-05-24: Last successful WAL-G basebackup wa... (gitlab-com/gl-infra/production#19883 - closed)
2025-05-24 : Private runners not picking up jobs (gitlab-com/gl-infra/production#19882 - closed)
2025-05-23 : Unable to view or create new issues (gitlab-com/gl-infra/production#19879 - closed)
2025-05-23: GitLab Job has failed (gitlab-com/gl-infra/production#19875 - closed)
2025-05-22 : Apdex SLO violation for workhorse ... (gitlab-com/gl-infra/production#19869 - closed)
2025-05-22 : 2025-05-22 | Data-Server Rebuild A... (gitlab-com/gl-infra/production#19868 - closed)
2025-05-22 : Error rate SLO violation in git se... (gitlab-com/gl-infra/production#19867 - closed)
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19859+
2025-05-20 : MR comments cannot load (gitlab-com/gl-infra/production#19855)
2025-05-19 : Failure of GCP scheduled snapshots... (gitlab-com/gl-infra/production#19852 - closed)
2025-05-19 : Error rate violation for atlantis_... (gitlab-com/gl-infra/production#19848 - closed)
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19810+
2025-06-03 - FY26 Q1 HAProxy Zonal Gameday - tm... (gitlab-com/gl-infra/production#19763 - closed)
Container Registry: Apply post-deployment migra... (gitlab-com/gl-infra/production#19671 - closed)
Draft: [GPRD] [2025-05-24 to 2025-05-28] - Upgr... (gitlab-com/gl-infra/production#19589 - closed)
[GSTG][Sec Decomp] - Phase 4 Rollout - Separate... (gitlab-com/gl-infra/production#19491 - closed)

Instructions

  • Review the Additional incidents list and add the Deploys-blocked and RootCause labels to all block deployments incidents. Due to a current limitation of incident.io, the labels added in advance are all removed.
  • Retrigger this pipeline job to update this report and metrics with the updated values. Consider this will discard any manual changes added.
  • Update the "weekly overview" table of this issue to also include:
    • any blocking CRs
  • Update the Deployments metric review epic.
    • Add a new row to the Overview section: Copy and paste the information from the Overview section in this issue and link to this issue in the Breakdown of blockers column.
    • Update the Graph: Update the data on the spreadsheet and then update the graph on the Deployments metric review epic.

📣 The deployment blockers per week can now be visualized in the Deployment Blockers Dashboard.

Edited by Dat Tang