Sidekiq exporter doesn't play well with multiple sidekiq-cluster instances
sidekiq-cluster spawns multiple Sidekiq processes, and each of them attempt to listen to the exporter port. Only one of them succeeds, and the others see:
2018-04-18_19:07:13.72861 sidekiq-besteffort-01 sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:205:in `bind': Address already in use - bind(2) for 0.0.0.0:8082 (Errno::EADDRINUSE)
2018-04-18_19:07:13.72868 sidekiq-besteffort-01 sidekiq-cluster: from /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:205:in `listen'
2018-04-18_19:07:13.72870 sidekiq-besteffort-01 sidekiq-cluster: from /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:759:in `block in tcp_server_sockets'
2018-04-18_19:07:13.72872 sidekiq-besteffort-01 sidekiq-cluster: from /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:231:in `each'
2018-04-18_19:07:13.72874 sidekiq-besteffort-01 sidekiq-cluster: from /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:231:in `foreach'
2018-04-18_19:07:13.73467 sidekiq-besteffort-01 sidekiq-cluster: from /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:757:in `tcp_server_sockets'
2018-04-18_19:07:13.73491 sidekiq-besteffort-01 sidekiq-cluster: from /opt/gitlab/embedded/lib/ruby/2.3.0/webrick/utils.rb:65:in `create_listeners'
2018-04-18_19:07:13.73518 sidekiq-besteffort-01 sidekiq-cluster: from /opt/gitlab/embedded/lib/ruby/2.3.0/webrick/server.rb:134:in `listen'
2018-04-18_19:07:13.73540 sidekiq-besteffort-01 sidekiq-cluster: from /opt/gitlab/embedded/lib/ruby/2.3.0/webrick/server.rb:115:in `initialize'
2018-04-18_19:07:13.73545 sidekiq-besteffort-01 sidekiq-cluster: from /opt/gitlab/embedded/lib/ruby/2.3.0/webrick/httpserver.rb:47:in `initialize'
2018-04-18_19:07:13.73549 sidekiq-besteffort-01 sidekiq-cluster: from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sidekiq_metrics_exporter.rb:27:in `new'
2018-04-18_19:07:13.73553 sidekiq-besteffort-01 sidekiq-cluster: from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sidekiq_metrics_exporter.rb:27:in `start_working'
2018-04-18_19:07:13.73561 sidekiq-besteffort-01 sidekiq-cluster: from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/daemon.rb:35:in `block (2 levels) in start'
We probably want to incorporate some instance ID/offset?
/cc: @joshlambert, @bjk-gitlab