Hanging specs when databases decomposed

These feature specs hangs on this MR:

• spec/features/merge_request/user_sees_pipelines_spec.rb - !80848 (merged)
• spec/features/admin/admin_runners_spec.rb
• spec/features/triggers_spec.rb
• spec/features/expand_collapse_diffs_spec.rb
• [-] ee/spec/features/registrations/combined_registration_spec.rb

Reproduction

$ export GITLAB_USE_MODEL_LOAD_BALANCING=true
$ export CACHE_CLASSES=true
$ export GITLAB_TEST_EAGER_LOAD=true
$ bin/rspec -f d spec/features/admin/admin_runners_spec.rb

**hangs**

At this stage start lldb on mac, or gdb

$ lldb
(lldb) process attach --pid 73448

# See https://saveriomiroddi.github.io/Debugging-a-live-stuck-ruby-process-with-gdb-fully-explained-plus-many-goodies/#other-interesting-sources
(lldb) call (void) rb_p((unsigned long) rb_eval_string_protect("Thread.list.each {|t| puts t; puts t.backtrace}", (int*)0))

Alternative install https://github.com/fluent/sigdump#install which will give you a nice dump of all Threads' backtraces.

Investigation

I confirmed that spec/features/merge_request/user_sees_pipelines_spec.rb is definitely hanging on the autoload lock (load_interlock_aware_monitor.rb)

• From a discussion: (+1 comment)

  Hang with spec/features/merge_request/user_sees_pipelines_spec.rb

  Found this piece of magic in the internet, which lists the backtrace of every Thread:

  call rb_p(rb_eval_string_protect("Thread.list.each {|t| puts t; puts t.backtrace.join(%Q[\n]); }", (int*)0))

  thread_backtraces.txt

  The backtraces is stuck in load_interlock_aware_monitor.rb:17

  Possibly relevant issues :

  • https://github.com/rails/rails/issues/37092
  • https://github.com/rails/rails/issues/27025
  • https://github.com/rails/rails/issues/34310

thread_backtraces.txt