Error UnicornSampler singleton instance already initialized
Summary
Gitlab fails with internal error induced by uncatched exception when configured with remote Sidekiq and prometheus metrics enabled.
Steps to reproduce
- Configure Gitlab unicorn with remote Sidekiq
- Make sure the
prometheus_metrics_enabled
is true:psql -U gitlab gitlabhq_production -c 'update application_settings set prometheus_metrics_enabled = true;'
- Try to login
What is the current bug behavior?
The unicorn returns HTTP 500 due to an uncatched exception.
What is the expected correct behavior?
User should successfully login and no exception should be thrown
Relevant logs and/or screenshots
RuntimeError (Gitlab::Metrics::Samplers::UnicornSampler singleton instance already initialized):
lib/gitlab/daemon.rb:6:in `initialize_instance'
config/initializers/7_prometheus_metrics.rb:33:in `block in <top (required)>'
lib/gitlab/cluster/lifecycle_events.rb:52:in `block in do_worker_start'
lib/gitlab/cluster/lifecycle_events.rb:51:in `each'
lib/gitlab/cluster/lifecycle_events.rb:51:in `do_worker_start'
lib/gitlab/middleware/rails_queue_duration.rb:24:in `call'
lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'
lib/gitlab/metrics/transaction.rb:55:in `run'
lib/gitlab/metrics/rack_middleware.rb:17:in `call'
lib/gitlab/middleware/multipart.rb:103:in `call'
lib/gitlab/request_profiler/middleware.rb:16:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
lib/gitlab/correlation_id.rb:15:in `use_id'
lib/gitlab/middleware/correlation_id.rb:15:in `call'
lib/gitlab/middleware/read_only/controller.rb:40:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/request_context.rb:20:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call'
lib/gitlab/middleware/release_env.rb:13:in `call'
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Current User: whoami: cannot find name for user ID 100002000 Using RVM: no Ruby Version: 2.5.3p105 Gem Version: 2.7.6 Bundler Version:1.16.6 Rake Version: 12.3.2 Redis Version: 3.2.12 Git Version: 2.18.1 Sidekiq Version:5.2.3 Go Version: unknown
GitLab information Version: 11.7.0 Revision: 1d9280e Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql URL: http://nginx:18080/gitlab HTTP Clone URL: http://nginx:18080/gitlab/some-group/some-project.git SSH Clone URL: git@nginx:some-group/some-project.git Using LDAP: yes Using Omniauth: yes Omniauth Providers: openid_connect
GitLab Shell Version: unknown Repository storage paths:
- default: /opt/test/data/gitlab/repositories Hooks: /opt/test/var/gitlab/hooks Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Warning You are running as user whoami: cannot find name for user ID 100002000, we hope you know what you are doing. Things may work/fail for the wrong reasons. For correct results you should run this as user git.Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 8.4.4 ? ... OK (8.4.4) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK
Access to /opt/test/shared/gitlab/authorized_keys: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... no Try fixing it: sudo -u git -H RAILS_ENV=production bin/background_jobs start For more information see: doc/install/installation.md in section "Install Init Script" see log/sidekiq.log for possible errors Please fix the error above and rerun the checks.
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... Server: ldapmain LDAP authentication... Success LDAP users with access to your GitLab server (only showing the first 100 results) DN: uid=dummy1,cn=users,cn=accounts,dc=example,dc=de uid: dummy1 DN: uid=dummy2,cn=users,cn=accounts,dc=example,dc=de uid: dummy2 DN: uid=dummy3,cn=users,cn=accounts,dc=example,dc=de uid: dummy3 DN: uid=dummy4,cn=users,cn=accounts,dc=example,dc=de uid: dummy1 DN: uid=dummyadmin,cn=users,cn=accounts,dc=example,dc=de uid: dummyadmin
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet) Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... can't check, you have no projects Redis version >= 2.8.0? ... yes Ruby version >= 2.3.5 ? ... yes (2.5.3) Git version >= 2.18.0 ? ... yes (2.18.1) Git user has default SSH configuration? ... yes Active users: ... 2
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
Possible fixes
The singleton implemented in lib/gitlab/daemon.rb
should be refactored to avoid throwing an exception when an instance was already initialized for a set of args and should just return the already existing instance.