Fix: security_orchestration_policy_spec.rb hangs on creating_policy_pipeline
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
Summary
The Developer Experience (DevEx) team is working on improving pipeline stability EPIC and has identified a recurring failure due to security_orchestration_policy_spec.rb That's causing significant job timeouts.
Problem Description
The spec file ee/spec/lib/ee/gitlab/ci/pipeline/chain/validate/security_orchestration_policy_spec.rb is expected to take a few minutes to run, but hangs for 60 + minutes before timing out.
Error Details
Here is the error message
Click to view Error logs
# [RSpecRunTime] Starting example group ee/spec/lib/ee/gitlab/ci/pipeline/chain/validate/security_orchestration_policy_spec.rb. Expected to take 1.34 seconds.
Gitlab::Ci::Pipeline::Chain::Validate::SecurityOrchestrationPolicy
#perform
when creating_policy_pipeline? is true
Terminated
WARNING: step_script could not run to completion because the timeout was exceeded. For more control over job and script timeouts see: https://docs.gitlab.com/ci/runners/configure_runners/#set-script-and-after_script-timeouts
ERROR: Job failed: execution took longer than 1h30m0s seconds
Impact
- Frequency: 14 occurrences in the last 28 days
- Duration: Jobs hang for 60+ minutes before timing out
Related Issue
gitlab-org/quality/analytics/team#145
Related Links
Here is the list of failed jobs, due to the same error
Click to view Failed JOb URL's
https://gitlab.com/gitlab-org/gitlab/-/jobs/10833696886
https://gitlab.com/gitlab-org/gitlab/-/jobs/10828634195
https://gitlab.com/gitlab-org/gitlab/-/jobs/10830135187
https://gitlab.com/gitlab-org/gitlab/-/jobs/10824554685
https://gitlab.com/gitlab-org/gitlab/-/jobs/10825593885
https://gitlab.com/gitlab-org/gitlab/-/jobs/10827313775
https://gitlab.com/gitlab-org/gitlab/-/jobs/10824529397
https://gitlab.com/gitlab-org/gitlab/-/jobs/10824412340
https://gitlab.com/gitlab-org/gitlab/-/jobs/10824561173
https://gitlab.com/gitlab-org/gitlab/-/jobs/10824334318
https://gitlab.com/gitlab-org/gitlab/-/jobs/10824198903