Gitlab::Daemon.stop fails during server shutdown

Gitlab::Daemon.stop uses mutex.synchronize.

mutex.synchronize cannot be called inside a signal handler, for reasons described in https://ruby-doc.org/core-trunk/doc/signals_rdoc.html

As with implementing signal handlers in C or most other languages, all code passed to Signal.trap must be reentrant. If you are not familiar with reentrancy, you need to read up on it at Wikipedia or elsewhere before reading the rest of this document.

Most importantly, “thread-safety” does not guarantee reentrancy; and methods such as Mutex#lock and Mutex#synchronize which are commonly used for thread-safety even prevent reentrancy.

I noticed this issue after redirecting puma logs to the GDK logs (see !19082 (merged))

/home/git/gitlab/lib/gitlab/daemon.rb:44:in `synchronize': can't be called from trap context (ThreadError)
	from /home/git/gitlab/lib/gitlab/daemon.rb:44:in `stop'
	from /home/git/gitlab/config/initializers/7_prometheus_metrics.rb:76:in `block in <main>'
	from /home/git/gitlab/lib/gitlab/cluster/lifecycle_events.rb:128:in `block in do_before_phased_restart'
	from /home/git/gitlab/lib/gitlab/cluster/lifecycle_events.rb:127:in `each'
	from /home/git/gitlab/lib/gitlab/cluster/lifecycle_events.rb:127:in `do_before_phased_restart'
	from /home/git/gitlab/lib/gitlab/cluster/mixins/puma_cluster.rb:12:in `stop_workers'
	from /Users/andrewn/.rvm/gems/ruby-2.6.3/gems/puma-3.12.0/lib/puma/cluster.rb:391:in `block in setup_signals'
	from /Users/andrewn/.rvm/gems/ruby-2.6.3/gems/puma-3.12.0/lib/puma/cluster.rb:488:in `select'
	from /Users/andrewn/.rvm/gems/ruby-2.6.3/gems/puma-3.12.0/lib/puma/cluster.rb:488:in `run'
	from /Users/andrewn/.rvm/gems/ruby-2.6.3/gems/puma-3.12.0/lib/puma/launcher.rb:184:in `run'
	from /Users/andrewn/.rvm/gems/ruby-2.6.3/gems/puma-3.12.0/lib/puma/cli.rb:78:in `run'
	from /Users/andrewn/.rvm/gems/ruby-2.6.3/gems/puma-3.12.0/bin/puma:10:in `<top (required)>'
	from /Users/andrewn/.rvm/gems/ruby-2.6.3/bin/puma:23:in `load'
	from /Users/andrewn/.rvm/gems/ruby-2.6.3/bin/puma:23:in `<main>'
	from /Users/andrewn/.rvm/gems/ruby-2.6.3/bin/ruby_executable_hooks:24:in `eval'
	from /Users/andrewn/.rvm/gems/ruby-2.6.3/bin/ruby_executable_hooks:24:in `<main>'

Whats happening?

On shutdown, Puma receives a signal, which is handled in puma-3.12.0/lib/puma/cluster.rb:391 in setup_signals. This calls do_before_phased_restart which in turn calls Gitlab::Metrics::Exporter::WebExporter.instance.stop in this block in config/initializers/7_prometheus_metrics.rb

  Gitlab::Cluster::LifecycleEvents.on_before_phased_restart do
    # We need to ensure that before we re-exec server
    # we do stop the exporter
    Gitlab::Metrics::Exporter::WebExporter.instance.stop
  end

However, since we're running in a signal handler, the stop method cannot call mutex.synchronize and raises can't be called from trap context (ThreadError).

cc @ayufan