Deployment Blockers - Week: 2024-06-24-2024-06-30

Overview

Start date End date Production deployments blocked for # of blockers
2024-06-24 2024-06-30 45.0 32

Failure types overview

Failure Type Time blocked on gstg Time blocked on gprd
RootCauseFlaky-Test 5.0 5.0
RootCauseSoftware-Change 39.0 39.0
RootCauseSaturation 1.0 1.0
Total 45.0 45.0

Weekly overview

Issue Blocker type gstg gprd
QA failures on gstg-cny (#11474 - closed) 0.0 0.0
QA failures on gprd-cny (#11468 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gprd-cny (#11467 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#11451 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#11446 - closed) 0.0 0.0
QA failures on gstg-cny (#11443 - closed) 0.0 0.0
QA failures on gstg-cny (#11441 - closed) 0.0 0.0
QA failure: qa/specs/features/browser_ui/9_data... (#11439 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/9_data... (#11438 - closed) 0.0 0.0
QA failure: qa/specs/features/browser_ui/9_data... (#11437 - closed) 0.0 0.0
QA failures on gstg-cny (#11431 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: (#11428 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#11413 - closed) RootCauseSoftware-Change 0.0 0.0
QA failures on gstg-cny (#11411 - closed) RootCauseSoftware-Change 0.0 0.0
QA failure: qa/specs/features/browser_ui/3_crea... (#11396 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: (#11393 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#11387 - closed) RootCauseSoftware-Change 0.0 0.0
Tuesday 2024-06-25 15:34 UTC - `gitlab-org/gitl... (#11385 - closed) 0.0 0.0
QA failures on gstg-cny (#11384 - closed) RootCauseSoftware-Change 0.0 0.0
QA failure: qa/specs/features/ee/browser_ui/10_... (#11381 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#11379 - closed) RootCauseSoftware-Change 0.0 0.0
QA failure: qa/specs/features/ee/api/2_plan/epi... (#11374 - closed) 0.0 0.0
Monday 2024-06-24 22:06 UTC - `gitlab-org/gitla... (#11372 - closed) 0.0 0.0
Monday 2024-06-24 22:02 UTC - `gitlab-org/gitla... (#11371 - closed) 0.0 0.0
Monday 2024-06-24 18:12 UTC - `gitlab-org/gitla... (#11366 - closed) 0.0 0.0
Monday 2024-06-24 17:28 UTC - `gitlab-org/gitla... (#11363 - closed) RootCauseExternal-Dependency 0.0 0.0
Monday 2024-06-24 17:17 UTC - `gitlab-org/gitla... (#11362 - closed) RootCauseExternal-Dependency 0.0 0.0
QA failures on gstg-cny (#11347 - closed) RootCauseFlaky-Test 0.5 0.5
2024-06-27: Deployment blocked due to bad line ... (gitlab-com/gl-infra/production#18206 - closed) RootCauseSoftware-Change 10.0 10.0
2024-06-27: Deployment blocked due to E2E failu... (gitlab-com/gl-infra/production#18205 - closed) RootCauseSoftware-Change 5.0 5.0
2024-06-25: qa image unable to be pulled from dev (gitlab-com/gl-infra/production#18193 - closed) RootCauseSaturation 1.0 1.0
2024-06-25: staging smoke tests failing - 500 o... (gitlab-com/gl-infra/production#18190 - closed) RootCauseSoftware-Change 24.0 24.0
Total 45.0 45.0

Additional incidents

Below is a list of production incidents created last week.

Click to expand
Issue
2024-06-30: Increase in graphql errors on cny (gitlab-com/gl-infra/production#18214 - closed)
2024-06-29: Zoekt persistent volume claim satur... (gitlab-com/gl-infra/production#18213 - closed)
2024-06-28: FrontendServiceWebsocketsServicesEr... (gitlab-com/gl-infra/production#18211 - closed)
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18207+
2024-06-27: Missing traffic on elasticsearch se... (gitlab-com/gl-infra/production#18203 - closed)
2024-06-27: AI Gateway inference error increase... (gitlab-com/gl-infra/production#18201 - closed)
2024-06-27: Large amount of queued sidekiq jobs... (gitlab-com/gl-infra/production#18200 - closed)
2024-06-26: Packages Error SLO Violation (gitlab-com/gl-infra/production#18197 - closed)
2024-06-26: CNY - API Graphql Error SLO (gitlab-com/gl-infra/production#18196 - closed)
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18191+
2024-06-25: DBRE project https://gitlab.com/git... (gitlab-com/gl-infra/production#18188 - closed)
2024-06-24: FrontendServiceWebsocketsServicesEr... (gitlab-com/gl-infra/production#18185 - closed)
2024-06-24: web error rate increase (cny) (gitlab-com/gl-infra/production#18182 - closed)

Instructions

  • Review the "Additional incidents" list and add the Deploys-blocked label if required.
  • Update the "weekly overview" table of this issue to also include:
    • any blocking CRs
    • the Root Cause type for each blocker (if this one is missing)
  • Update the Deployments metric review epic.
    • Add a new row to the Overview section: Copy and paste the information from the Overview section in this issue and link to this issue in the Breakdown of blockers column.
    • Update the Graph: Update the data on the spreadsheet and then update the graph on the Deployments metric review epic.
Edited Jul 03, 2024 by Jenny Kim
Assignee Loading
Time tracking Loading