Deployment Blockers - Week: 2024-06-17-2024-06-23

Overview

Start date End date Production deployments blocked for # of blockers
2024-06-17 2024-06-23 21 28

Failure types overview

Failure Type Time blocked on gstg Time blocked on gprd
RootCauseFlaky-Test 6.0 6.0
RootCauseSaturation 3.0 3.0
RootCauseNeeded 2.0 2.0
RootCauseSoftware-Change 8.0 8.0
RootCauseDB-Migration 1.0 1.0
RootCauseExternal-Dependency 1.0 1.0
Total 21 21

Weekly overview

Issue Blocker type gstg gprd
QA failure: qa/specs/features/browser_ui/3_crea... (#11340 - closed) RootCauseFlaky-Test 0.5 0.5
Friday 2024-06-21 18:07 UTC - `gitlab-org/gitla... (#11336 - closed) RootCauseFlaky-Test 0.0 0.0
QA failure: qa/specs/features/browser_ui/5_pack... (#11320 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#11311 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: (#11309 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/ee/browser_ui/3_c... (#11304 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#11302 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/ee/browser_ui/3_c... (#11295 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#11288 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#11282 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#11279 - closed) Pipeline skipped 0.0 0.0
QA failure: qa/specs/features/ee/browser_ui/10_... (#11273 - closed) Pipeline skipped 0.0 0.0
QA failure: qa/specs/features/browser_ui/3_crea... (#11266 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gprd-cny (#11264 - closed) RootCauseFlaky-Test 0.0 0.0
Tuesday 2024-06-18 11:13 UTC - `gitlab-org/gitl... (#11256 - closed) RootCauseIndeterminate 0.0 0.0
QA failure: qa/specs/features/api/4_verify/file... (#11246 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/4_veri... (#11238 - closed) RootCauseFlaky-Test 0.5 0.5
Monday 2024-06-17 17:40 UTC - `gitlab-org/gitla... (#11237 - closed) RootCauseFlaky-Test 0.0 0.0
Monday 2024-06-17 17:36 UTC - `gitlab-org/gitla... (#11236 - closed) RootCauseFlaky-Test 0.0 0.0
Monday 2024-06-17 17:18 UTC - `gitlab-org/gitla... (#11234 - closed) RootCauseFlaky-Test 0.0 0.0
Monday 2024-06-17 17:02 UTC - `gitlab-org/gitla... (#11233 - closed) RootCauseFlaky-Test 0.0 0.0
Monday 2024-06-17 16:50 UTC - `gitlab-org/gitla... (#11232 - closed) Flaky job 0.0 0.0
Monday 2024-06-17 16:36 UTC - `gitlab-org/gitla... (#11231 - closed) RootCauseFlaky-Test 0.0 0.0
2024-06-18: pgbouncer sidekiq saturation (gitlab-com/gl-infra/production#18167 - closed) RootCauseSaturation 3.0 3.0
2024-06-18: gprd-gitaly job failed due to unrea... (gitlab-com/gl-infra/production#18165 - closed) RootCauseNeeded 2.0 2.0
2024-06-17: Migration job failing on gstg-cny (gitlab-com/gl-infra/production#18155 - closed) RootCauseSoftware-Change 8.0 8.0
2024-06-21: PG transaction deadlock (gitlab-com/gl-infra/production#18176 - closed) RootCauseDB-Migration 1.0 1.0
2024-06-20: 502 Bad Gateway on gitlab.com (gitlab-com/gl-infra/production#18175 - closed) RootCauseExternal-Dependency 1.0 1.0
Total 21 21

Additional incidents

Below is a list of production incidents created last week.

Click to expand
Issue
2024-06-22: WebServiceRailsRequestErrorSLOViola... (gitlab-com/gl-infra/production#18179 - closed)
2024-06-21: PG transaction deadlock (gitlab-com/gl-infra/production#18176 - closed)
2024-06-20: 502 Bad Gateway on gitlab.com (gitlab-com/gl-infra/production#18175 - closed)
2024-06-20: Uptick in 429 errors, unexpected au... (gitlab-com/gl-infra/production#18173 - closed)
2024-06-19: GCPScheduledSnapshotsFailed patroni... (gitlab-com/gl-infra/production#18171 - closed)
2024-06-19: MacOS runners are not picking up jobs (gitlab-com/gl-infra/production#18170 - closed)
2024-06-19: Patroni Consul service has more tha... (gitlab-com/gl-infra/production#18168 - closed)
2024-06-18: Short apdex dip on kube apiserver (gitlab-com/gl-infra/production#18162 - closed)
2024-06-18: n2d CPU quota reached for gitlab-ci... (gitlab-com/gl-infra/production#18160 - closed)
2024-06-17: gitlab-org/gitlab pipelines get `fa... (gitlab-com/gl-infra/production#18156 - closed)

Instructions

  • Review the "Additional incidents" list and add the Deploys-blocked label if required.
  • Update the "weekly overview" table of this issue to also include:
    • any blocking CRs
    • the Root Cause type for each blocker (if this one is missing)
  • Update the Deployments metric review epic.
    • Add a new row to the Overview section: Copy and paste the information from the Overview section in this issue and link to this issue in the Breakdown of blockers column.
    • Update the Graph: Update the data on the spreadsheet and then update the graph on the Deployments metric review epic.
Edited by Jenny Kim