Hanging specs when databases decomposed
These feature specs hangs on this MR:
-
spec/features/merge_request/user_sees_pipelines_spec.rb - !80848 (merged) -
spec/features/admin/admin_runners_spec.rb -
spec/features/triggers_spec.rb -
spec/features/expand_collapse_diffs_spec.rb - [-]
ee/spec/features/registrations/combined_registration_spec.rb
Reproduction
$ export GITLAB_USE_MODEL_LOAD_BALANCING=true
$ export CACHE_CLASSES=true
$ export GITLAB_TEST_EAGER_LOAD=true
$ bin/rspec -f d spec/features/admin/admin_runners_spec.rb
**hangs**
At this stage start lldb
on mac, or gdb
$ lldb
(lldb) process attach --pid 73448
# See https://saveriomiroddi.github.io/Debugging-a-live-stuck-ruby-process-with-gdb-fully-explained-plus-many-goodies/#other-interesting-sources
(lldb) call (void) rb_p((unsigned long) rb_eval_string_protect("Thread.list.each {|t| puts t; puts t.backtrace}", (int*)0))
Alternative install https://github.com/fluent/sigdump#install which will give you a nice dump of all Threads' backtraces.
Investigation
I confirmed that spec/features/merge_request/user_sees_pipelines_spec.rb is definitely hanging on the autoload lock (load_interlock_aware_monitor.rb)
-
From a discussion: (+1 comment)
Found this piece of magic in the internet, which lists the backtrace of every Thread:
call rb_p(rb_eval_string_protect("Thread.list.each {|t| puts t; puts t.backtrace.join(%Q[\n]); }", (int*)0))
The backtraces is stuck in
load_interlock_aware_monitor.rb:17
Possibly relevant issues :