Prometheus client mmap: exception data slice is nil
While investigating https://gitlab.com/gitlab-org/gitlab-ce/issues/45281, we see a lot of these in the unicorn stderr logs:
E, [2018-04-13T15:17:22.888837 #48339] ERROR -- : exception data slice is nil at pos 16376 while processing metrics file /run/gitlab/unicorn/counter_worker_id_6-0.db (Prometheus::Client::ParsingError)
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/prometheus-client-mmap-0.9.1/lib/prometheus/client/mmaped_dict.rb:38:in `rescue in initialize'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/prometheus-client-mmap-0.9.1/lib/prometheus/client/mmaped_dict.rb:27:in `initialize'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/prometheus-client-mmap-0.9.1/lib/prometheus/client/mmaped_value.rb:117:in `new'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/prometheus-client-mmap-0.9.1/lib/prometheus/client/mmaped_value.rb:117:in `unsafe_initialize_file'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/prometheus-client-mmap-0.9.1/lib/prometheus/client/mmaped_value.rb:62:in `unsafe_reinitialize_file'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/prometheus-client-mmap-0.9.1/lib/prometheus/client/mmaped_value.rb:87:in `each_object'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/prometheus-client-mmap-0.9.1/lib/prometheus/client/mmaped_value.rb:87:in `block in reinitialize_on_pid_change'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/prometheus-client-mmap-0.9.1/lib/prometheus/client/mmaped_value.rb:84:in `synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/prometheus-client-mmap-0.9.1/lib/prometheus/client/mmaped_value.rb:84:in `reinitialize_on_pid_change'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/prometheus-client-mmap-0.9.1/lib/prometheus/client.rb:33:in `reinitialize_on_pid_change'
/var/opt/gitlab/gitlab-rails/etc/unicorn.rb:40:in `block in reload'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:611:in `init_worker_process'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:639:in `worker_loop'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:519:in `maintain_worker_count'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:283:in `join'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
/cc: @bjk-gitlab, @pchojnacki
Edited by Stan Hu