Skip to content

Fix: security_orchestration_policy_spec.rb hangs on creating_policy_pipeline

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Summary

The Developer Experience (DevEx) team is working on improving pipeline stability EPIC and has identified a recurring failure due to security_orchestration_policy_spec.rb That's causing significant job timeouts.

Problem Description

The spec file ee/spec/lib/ee/gitlab/ci/pipeline/chain/validate/security_orchestration_policy_spec.rb is expected to take a few minutes to run, but hangs for 60 + minutes before timing out.

Error Details

Here is the error message

Click to view Error logs
# [RSpecRunTime] Starting example group ee/spec/lib/ee/gitlab/ci/pipeline/chain/validate/security_orchestration_policy_spec.rb. Expected to take 1.34 seconds.
Gitlab::Ci::Pipeline::Chain::Validate::SecurityOrchestrationPolicy
  #perform
    when creating_policy_pipeline? is true
Terminated
WARNING: step_script could not run to completion because the timeout was exceeded. For more control over job and script timeouts see: https://docs.gitlab.com/ci/runners/configure_runners/#set-script-and-after_script-timeouts
ERROR: Job failed: execution took longer than 1h30m0s seconds

Impact

  • Frequency: 14 occurrences in the last 28 days
  • Duration: Jobs hang for 60+ minutes before timing out

Related Issue

gitlab-org/quality/analytics/team#145

Related Links

Here is the list of failed jobs, due to the same error

Click to view Failed JOb URL's

https://gitlab.com/gitlab-org/gitlab/-/jobs/10833696886

https://gitlab.com/gitlab-org/gitlab/-/jobs/10828634195

https://gitlab.com/gitlab-org/gitlab/-/jobs/10830135187

https://gitlab.com/gitlab-org/gitlab/-/jobs/10824554685

https://gitlab.com/gitlab-org/gitlab/-/jobs/10825593885

https://gitlab.com/gitlab-org/gitlab/-/jobs/10827313775

https://gitlab.com/gitlab-org/gitlab/-/jobs/10824529397

https://gitlab.com/gitlab-org/gitlab/-/jobs/10824412340

https://gitlab.com/gitlab-org/gitlab/-/jobs/10824561173

https://gitlab.com/gitlab-org/gitlab/-/jobs/10824334318

https://gitlab.com/gitlab-org/gitlab/-/jobs/10824198903

https://gitlab.com/gitlab-org/gitlab/-/jobs/10824194420

https://gitlab.com/gitlab-org/gitlab/-/jobs/10824497299

Edited by 🤖 GitLab Bot 🤖