spec/tasks/gitlab/check_rake_spec.rb hanging and causing CI jobs to timeout
Problem
There are jobs either timing out on spec/tasks/gitlab/check_rake_spec.rb
or contains spec/tasks/gitlab/check_rake_spec.rb
which depletes most of the time permitted for a CI job (90mins).
Some affected CI jobs
- https://gitlab.com/gitlab-org/gitlab/-/jobs/5556344553
- https://gitlab.com/gitlab-org/gitlab/-/jobs/5553855426
Investigation notes
Using a temporary branch (!137138 (closed)), the cause of timeout has been narrowed down to the SystemCheck::App::MigrationsAreUpCheck
in https://gitlab.com/gitlab-org/gitlab/-/blob/f6c33e97b1961f7a4ab89322b54f3e7bb59bc250/lib/system_check/rake_task/app_task.rb#L17
Reproducing this issue
You can local reproduce this by running a background migration test before the check_rake_spec.rb
. For now this combination (derived from https://gitlab.com/gitlab-org/gitlab/-/jobs/5553855426) seems to consistently replicate the issue
FOSS_ONLY=1 bin/rspec -f d spec/lib/gitlab/background_migration/backfill_prepared_at_merge_requests_spec.rb spec/tasks/gitlab/check_rake_spec.rb
Edited by Sylvester Chin