Fix: Jest Vue3 MR and jest vue3 predictive jobs taking 80 minutes

Summary

Jest Vue3 mr and jest vue3 predictive Jobs are the largest contributor to job timeout failures in our CI pipeline, accounting for 46% of all detected hang time over the last 28 days.

Jobs consistently hang after completing test execution, specifically after the ai_genie_spec.js test passes.

This has resulted in 453+ hours of wasted CI compute time over the analyzed period, affecting 315 jobs with an average hang duration of ~86-88 minutes per job.

Problem Description

Impact

Metric Value
Total hang duration (jest vue3 mr) 438 hours 42 minutes
Total hang duration (jest vue3 predictive) 14 hours 44 minutes
Affected jobs (analyzed period) 315 jobs
Average hang duration per job ~86-88 minutes
Percentage of total CI hang time 81%

This represents significant wasted compute resources, delayed pipeline feedback, and increased infrastructure costs.

Hang Pattern

All affected jobs exhibit a consistent pattern:

  1. Tests execute and pass successfully
  2. Last logged message is typically: PASS ee/spec/frontend/ai/components/ai_genie_spec.js (XXX MB heap size)
  3. Job hangs for ~86-88 minutes until timeout is reached
  4. No further output or activity during hang period

Error Details

Job Statistics

Job Name Total Hang Duration Count
jest vue3 mr 26322.45 minutes (438 hrs 42 min) 305
jest vue3 predictive 884.35 minutes (14 hrs 44 min) 10

Sample Affected Jobs

Job Type Job ID Hang Duration Heap Size Last Message
jest vue3 predictive 12149287148 88.74 min N/A (Use node --trace-deprecation ...)
jest vue3 mr 12166097385 85.81 min 672 MB PASS ai_genie_spec.js
jest vue3 mr 12175618881 86.55 min 588 MB PASS ai_genie_spec.js
jest vue3 mr 12160013639 86.47 min 708 MB PASS ai_genie_spec.js
jest vue3 mr 12158381842 86.22 min 776 MB PASS ai_genie_spec.js
jest vue3 mr 12155310480 86.85 min 653 MB PASS ai_genie_spec.js
jest vue3 mr 12154918461 86.77 min 585 MB PASS ai_genie_spec.js
jest vue3 mr 12152604402 86.50 min 674 MB PASS ai_genie_spec.js
jest vue3 mr 12152244904 86.74 min 759 MB PASS ai_genie_spec.js
jest vue3 mr 12150033377 86.90 min 608 MB PASS ai_genie_spec.js
Additional affected jobs (click to expand)

Jest Vue3 Predictive Jobs:

Jest Vue3 MR Jobs (sample - 305 total):

Key Observations

  1. Consistent timing: Hangs are almost always between 85-89 minutes, indicating jobs are hitting the timeout limit rather than hanging at varying points
  2. Post-completion hang: The hang occurs after test completion, not during test execution
  3. ai_genie_spec.js correlation: This spec consistently appears as the last passing test before the hang
  4. Node deprecation warnings: Some jobs show (Use node --trace-deprecation ...) as the last message.

Related Issue

gitlab-org/quality/analytics/team#145

Edited by 🤖 GitLab Bot 🤖