GitLab sometimes doesn't start - fails to start a new unicorn master because it's already running
Summary
Sometimes GitLab (12.5.3 225d2e5bb89) fails to start and gets stuck in the infinite loop described below. If I nuke the container (official Omnibus) and get a new one (pointing at the same data,log,etc. directories) it starts working again.
Steps to reproduce
Unfortunately I don't have a way of reliably reproducing this, but it has happened multiple times over the past few weeks.
Relevant logs and/or screenshots
This, many times over:
==> /var/log/gitlab/unicorn/current <==
2019-12-07_20:48:25.88936 failed to start a new unicorn master
2019-12-07_20:48:25.90141 starting new unicorn master
2019-12-07_20:48:26.37624 master failed to start, check stderr log for details
==> /var/log/gitlab/unicorn/unicorn_stderr.log <==
ArgumentError: Already running on PID:364 (or pid=/opt/gitlab/var/unicorn/unicorn.pid is stale)
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:205:in `pid='
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:137:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/bin/unicorn:126:in `<top (required)>'
/opt/gitlab/embedded/bin/unicorn:23:in `load'
/opt/gitlab/embedded/bin/unicorn:23:in `<top (required)>'