Deployment Blockers - Week: 2024-06-03-2024-06-09

Overview

Start date End date Production deployments blocked for # of blockers
2024-06-03 2024-06-09 42.0 27

Failure types overview

Failure Type Time blocked on gstg Time blocked on gprd
RootCauseFlaky-Test 6.5 6.5
RootCauseSoftware-Change 33.0 33.0
RootCauseNeeded 2.0 2.0
Total 41.5 41.5

Weekly overview

Issue Blocker type gstg gprd
QA failure: qa/specs/features/browser_ui/3_crea... (#11111 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/5_pack... (#11106 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/ee/browser_ui/10_... (#11095 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/2_plan... (#11093 - closed) 0.0 0.0
QA failure: (#11087 - closed) 0.0 0.0
QA failures on gstg-cny (#11084 - closed) 0.0 0.0
QA failures on gstg-cny (#11070 - closed) 0.0 0.0
Wednesday 2024-06-05 22:35 UTC - `gitlab-org/gi... (#11068 - closed) 0.0 0.0
QA failures on gstg-cny (#11067 - closed) 0.0 0.0
QA failures on gstg-cny (#11064 - closed) 0.0 0.0
QA failures on gstg-cny (#11063 - closed) 0.0 0.0
TEST - QA failure (#11051 - closed) 0.5 0.5
TEST QA failure: qa/specs/features/browser_ui/4... (#11050 - closed) 0.5 0.0
Tuesday 2024-06-04 17:53 UTC - `gitlab-org/gitl... (#11049 - closed) 0.0 0.0
QA failure: qa/specs/features/browser_ui/4_veri... (#11046 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#11044 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#11041 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#11032 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/ee/browser_ui/3_c... (#11029 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/3_crea... (#11024 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/5_pack... (#11019 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#11012 - closed) RootCauseFlaky-Test 0.5 0.5
QA failure: qa/specs/features/browser_ui/2_plan... (#11010 - closed) RootCauseFlaky-Test 0.5 0.5
QA failures on gstg-cny (#11008 - closed) RootCauseFlaky-Test 0.5 0.5
2024-06-06: deployment qa jobs failing to pull ... (gitlab-com/gl-infra/production#18126 - closed) RootCauseSoftware-Change 1.0 1.0
2024-06-05: Pipelines on dev projects are stuck (gitlab-com/gl-infra/production#18114 - closed) RootCauseNeeded 2.0 2.0
2024-06-05: page styling broken in staging-cana... (gitlab-com/gl-infra/production#18107 - closed) RootCauseSoftware-Change 32.0 32.0
Total 42.5 42.0

Additional incidents

Below is a list of production incidents created last week.

Click to expand
Issue
2024-06-07: Dead mans snitch: missing Alertmana... (gitlab-com/gl-infra/production#18131 - closed)
2024-06-07: Data-Server Rebuild Ansible | Faile... (gitlab-com/gl-infra/production#18130 - closed)
2024-06-06: Long-running transactions detected ... (gitlab-com/gl-infra/production#18125 - closed)
2024-06-06: Long-running transactions detected ... (gitlab-com/gl-infra/production#18124 - closed)
2024-06-06: Chef client has been disabled for a... (gitlab-com/gl-infra/production#18123 - closed)
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18122+
2024-06-06: Large amount of ES sidekiq jobs (gitlab-com/gl-infra/production#18121 - closed)
2024-06-06: FrontendServiceWebsocketsServicesEr... (gitlab-com/gl-infra/production#18120 - closed)
2024-06-06: Align file_size_limit_mb between ul... (gitlab-com/gl-infra/production#18119 - closed)
2024-06-06: Nat Gateway Port Allocation Saturation (gitlab-com/gl-infra/production#18118 - closed)
2024-06-05: PostgresSQL CommitRateTooLow (gitlab-com/gl-infra/production#18111 - closed)
2024-06-05: Alertmanager not recieving notifica... (gitlab-com/gl-infra/production#18110 - closed)
2024-06-05: TrafficAbsent alerts everywhere (gitlab-com/gl-infra/production#18109 - closed)
2024-06-05: redis-pubsub elevated error rate (gitlab-com/gl-infra/production#18108 - closed)
2024-06-04: AiGatewayServiceRunwayIngressApdexS... (gitlab-com/gl-infra/production#18105 - closed)
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18103+
2024-06-03: ElasticsearchHttp -- Exception: Net... (gitlab-com/gl-infra/production#18097 - closed)
2024-06-03: Long-running transactions detected ... (gitlab-com/gl-infra/production#18096 - closed)

Instructions

  • Review the "Additional incidents" list and add the Deploys-blocked label if required.
  • Update the "weekly overview" table of this issue to also include:
    • any blocking CRs
    • the Root Cause type for each blocker (if this one is missing)
  • Update the Deployments metric review epic.
    • Add a new row to the Overview section: Copy and paste the information from the Overview section in this issue and link to this issue in the Breakdown of blockers column.
    • Update the Graph: Update the data on the spreadsheet and then update the graph on the Deployments metric review epic.
Edited Jun 10, 2024 by Mayra Cabrera
Assignee Loading
Time tracking Loading