app error: deadlock; recursive locking

Summary

Steps to reproduce

Pending, as yet this has not re-occurred.

Example Project

What is the current bug behavior?

After the deadlock, the node was deleted.

What is the expected correct behavior?

There is no deadlock

Relevant logs and/or screenshots

E, [2020-03-05T02:43:36.617340 #32625] ERROR -- : app error: deadlock; recursive locking (ThreadError)
E, [2020-03-05T02:43:36.617387 #32625] ERROR -- : /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/lock.rb:13:in `lock'
E, [2020-03-05T02:43:36.617401 #32625] ERROR -- : /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/lock.rb:13:in `call'
E, [2020-03-05T02:43:36.617411 #32625] ERROR -- : /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/requests_rack_middleware.rb:49:in `call'
E, [2020-03-05T02:43:36.617422 #32625] ERROR -- : /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/sendfile.rb:111:in `call'
E, [2020-03-05T02:43:36.617440 #32625] ERROR -- : /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sentry-raven-2.9.0/lib/raven/integrations/rack.rb:51:in `call'
E, [2020-03-05T02:43:36.617450 #32625] ERROR -- : /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/engine.rb:524:in `call'
E, [2020-03-05T02:43:36.617458 #32625] ERROR -- : /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/railtie.rb:190:in `public_send'
E, [2020-03-05T02:43:36.617469 #32625] ERROR -- : /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/railtie.rb:190:in `method_missing'
E, [2020-03-05T02:43:36.617478 #32625] ERROR -- : /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/release_env.rb:12:in `call'
E, [2020-03-05T02:43:36.617487 #32625] ERROR -- : /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:68:in `block in call'
E, [2020-03-05T02:43:36.617496 #32625] ERROR -- : /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:53:in `each'
E, [2020-03-05T02:43:36.617511 #32625] ERROR -- : /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:53:in `call'
E, [2020-03-05T02:43:36.617520 #32625] ERROR -- : /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:606:in `process_client'
E, [2020-03-05T02:43:36.617530 #32625] ERROR -- : /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-worker-killer-0.4.4/lib/unicorn/worker_killer.rb:52:in `process_client'
E, [2020-03-05T02:43:36.617542 #32625] ERROR -- : /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:701:in `worker_loop'
E, [2020-03-05T02:43:36.617551 #32625] ERROR -- : /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:549:in `spawn_missing_workers'
E, [2020-03-05T02:43:36.617561 #32625] ERROR -- : /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:563:in `maintain_worker_count'
E, [2020-03-05T02:43:36.617571 #32625] ERROR -- : /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:293:in `join'
E, [2020-03-05T02:43:36.617579 #32625] ERROR -- : /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/bin/unicorn:126:in `<top (required)>'
E, [2020-03-05T02:43:36.617588 #32625] ERROR -- : /opt/gitlab/embedded/bin/unicorn:23:in `load'
E, [2020-03-05T02:43:36.617596 #32625] ERROR -- : /opt/gitlab/embedded/bin/unicorn:23:in `<top (required)>'
E, [2020-03-05T02:43:36.617606 #32625] ERROR -- : /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `load'
E, [2020-03-05T02:43:36.617615 #32625] ERROR -- : /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `kernel_load'
E, [2020-03-05T02:43:36.617624 #32625] ERROR -- : /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:28:in `run'
E, [2020-03-05T02:43:36.617634 #32625] ERROR -- : /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:463:in `exec'
E, [2020-03-05T02:43:36.617644 #32625] ERROR -- : /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
E, [2020-03-05T02:43:36.617652 #32625] ERROR -- : /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
E, [2020-03-05T02:43:36.617661 #32625] ERROR -- : /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
E, [2020-03-05T02:43:36.617670 #32625] ERROR -- : /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:27:in `dispatch'
E, [2020-03-05T02:43:36.617680 #32625] ERROR -- : /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
E, [2020-03-05T02:43:36.617689 #32625] ERROR -- : /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:18:in `start'
E, [2020-03-05T02:43:36.617701 #32625] ERROR -- : /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:30:in `block in <top (required)>'
E, [2020-03-05T02:43:36.617714 #32625] ERROR -- : /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
E, [2020-03-05T02:43:36.617723 #32625] ERROR -- : /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:22:in `<top (required)>'
E, [2020-03-05T02:43:36.617732 #32625] ERROR -- : /opt/gitlab/embedded/bin/bundle:23:in `load'
E, [2020-03-05T02:43:36.617741 #32625] ERROR -- : /opt/gitlab/embedded/bin/bundle:23:in `<main>'

Output of checks

Bug occurred on a self_hosted instance on AWS

Results of GitLab environment info

https://gitlab.zendesk.com/agent/tickets/149332 (Internal) Diagram is linked to org from the ticket

Dual zone S3 build, with nodes running specific services

Results of GitLab application Check

Expand for output related to the GitLab application check

(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)
(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)
(we will only investigate if the tests are passing)

Possible fixes

(If you can, link to the line of code that might be responsible for the problem)