After a 'Can't reserve bytes for memory-mapped file', Unicorn starts to throw 'unmapped file' and 'deadlock; recursive locking' errors

It appears that failed allocations cause files to get into a bad state.

After the Can't reserve 1048576 bytes for memory-mapped file error occurs, an unmapped file error is thrown shortly afterwards, failing on sync. After this, Unicorn workers start throwing deadlock; recursive locking errors repeatedly until GitLab is restarted.

The reporting customer was running into the deadlock errors after the now fixed SIGBUS error on allocation failure, as well.

Version: v0.10.0 (GitLab Omnibus Docker v12.7.5-ee)

Initial allocation failure

app error: Can't reserve 1048576 bytes for memory-mapped file (IOError)
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/prometheus-client-mmap-0.10.0/lib/prometheus/client/helper/mmaped_file.rb:21:in `initialize'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/prometheus-client-mmap-0.10.0/lib/prometheus/client/helper/mmaped_file.rb:21:in `initialize'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/prometheus-client-mmap-0.10.0/lib/prometheus/client/helper/mmaped_file.rb:38:in `new'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/prometheus-client-mmap-0.10.0/lib/prometheus/client/helper/mmaped_file.rb:38:in `open'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/prometheus-client-mmap-0.10.0/lib/prometheus/client/helper/mmaped_file.rb:50:in `open_exclusive_file'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/prometheus-client-mmap-0.10.0/lib/prometheus/client/mmaped_value.rb:115:in `unsafe_initialize_file'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/prometheus-client-mmap-0.10.0/lib/prometheus/client/mmaped_value.rb:103:in `block in initialize_file'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/prometheus-client-mmap-0.10.0/lib/prometheus/client/mmaped_value.rb:102:in `synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/prometheus-client-mmap-0.10.0/lib/prometheus/client/mmaped_value.rb:102:in `initialize_file'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/prometheus-client-mmap-0.10.0/lib/prometheus/client/mmaped_value.rb:31:in `block in increment'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/prometheus-client-mmap-0.10.0/lib/prometheus/client/mmaped_value.rb:30:in `synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/prometheus-client-mmap-0.10.0/lib/prometheus/client/mmaped_value.rb:30:in `increment'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/prometheus-client-mmap-0.10.0/lib/prometheus/client/counter.rb:17:in `block in increment'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/prometheus-client-mmap-0.10.0/lib/prometheus/client/metric.rb:76:in `synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/prometheus-client-mmap-0.10.0/lib/prometheus/client/metric.rb:76:in `synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/prometheus-client-mmap-0.10.0/lib/prometheus/client/counter.rb:17:in `increment'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/requests_rack_middleware.rb:47:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/middleware/host_authorization.rb:77:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sentry-raven-2.9.0/lib/raven/integrations/rack.rb:51:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/engine.rb:526:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/railtie.rb:190:in `public_send'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/railtie.rb:190:in `method_missing'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/release_env.rb:12:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:68:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:53:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:53:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:606:in `process_client'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-worker-killer-0.4.4/lib/unicorn/worker_killer.rb:52:in `process_client'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:701:in `worker_loop'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:549:in `spawn_missing_workers'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:563:in `maintain_worker_count'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:293:in `join'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/bin/unicorn:126:in `<top (required)>'
/opt/gitlab/embedded/bin/unicorn:23:in `load'
/opt/gitlab/embedded/bin/unicorn:23:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `load'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:28:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:463:in `exec'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:27:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:18:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:30:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:22:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Second error

app error: unmapped file (IOError)
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/prometheus-client-mmap-0.10.0/lib/prometheus/client/mmaped_dict.rb:54:in `sync'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/prometheus-client-mmap-0.10.0/lib/prometheus/client/mmaped_dict.rb:54:in `close'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/prometheus-client-mmap-0.10.0/lib/prometheus/client/mmaped_value.rb:113:in `unsafe_initialize_file'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/prometheus-client-mmap-0.10.0/lib/prometheus/client/mmaped_value.rb:103:in `block in initialize_file'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/prometheus-client-mmap-0.10.0/lib/prometheus/client/mmaped_value.rb:102:in `synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/prometheus-client-mmap-0.10.0/lib/prometheus/client/mmaped_value.rb:102:in `initialize_file'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/prometheus-client-mmap-0.10.0/lib/prometheus/client/mmaped_value.rb:31:in `block in increment'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/prometheus-client-mmap-0.10.0/lib/prometheus/client/mmaped_value.rb:30:in `synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/prometheus-client-mmap-0.10.0/lib/prometheus/client/mmaped_value.rb:30:in `increment'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/prometheus-client-mmap-0.10.0/lib/prometheus/client/histogram.rb:26:in `observe'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/prometheus-client-mmap-0.10.0/lib/prometheus/client/histogram.rb:65:in `block in observe'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/prometheus-client-mmap-0.10.0/lib/prometheus/client/metric.rb:76:in `synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/prometheus-client-mmap-0.10.0/lib/prometheus/client/metric.rb:76:in `synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/prometheus-client-mmap-0.10.0/lib/prometheus/client/histogram.rb:65:in `observe'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/requests_rack_middleware.rb:52:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/middleware/host_authorization.rb:77:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sentry-raven-2.9.0/lib/raven/integrations/rack.rb:51:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/engine.rb:526:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/railtie.rb:190:in `public_send'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/railtie.rb:190:in `method_missing'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/release_env.rb:12:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:68:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:53:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:53:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:606:in `process_client'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-worker-killer-0.4.4/lib/unicorn/worker_killer.rb:52:in `process_client'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:701:in `worker_loop'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:549:in `spawn_missing_workers'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:563:in `maintain_worker_count'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:293:in `join'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/bin/unicorn:126:in `<top (required)>'
/opt/gitlab/embedded/bin/unicorn:23:in `load'
/opt/gitlab/embedded/bin/unicorn:23:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `load'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:28:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:463:in `exec'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:27:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:18:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:30:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:22:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

These start occurring several times a second

app error: deadlock; recursive locking (ThreadError)
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/lock.rb:13:in `lock'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/lock.rb:13:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/sendfile.rb:111:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/requests_rack_middleware.rb:49:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/middleware/host_authorization.rb:77:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sentry-raven-2.9.0/lib/raven/integrations/rack.rb:51:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/engine.rb:526:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/railtie.rb:190:in `public_send'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/railtie.rb:190:in `method_missing'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/release_env.rb:12:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:68:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:53:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:53:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:606:in `process_client'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-worker-killer-0.4.4/lib/unicorn/worker_killer.rb:52:in `process_client'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:701:in `worker_loop'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:549:in `spawn_missing_workers'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:563:in `maintain_worker_count'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:293:in `join'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/bin/unicorn:126:in `<top (required)>'
/opt/gitlab/embedded/bin/unicorn:23:in `load'
/opt/gitlab/embedded/bin/unicorn:23:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `load'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:28:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:463:in `exec'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:27:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:18:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:30:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:22:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

/cc @stanhu