Skip to content

Test failures due to inability to reset Prometheus metrics

https://gitlab.com/gitlab-org/gitlab-ee/-/jobs/39609213 failed when I ran two specs that created Prometheus metrics:

bundle exec rspec spec/workers/geo/metrics_update_worker_spec.rb spec/services/geo/metrics_update_service_spec.rb

The problem is that gauge entries can only be appended to the .db file, and there's no easy way to reset them between test runs.

When I attempted to remove the .db files after the test and reset the registry, I got an ugly seg fault:

== Seed from /Users/stanhu/gitlab/gdk-ee/gitlab/db/fixtures/test/01_plan.rb
..
== Seed from /Users/stanhu/gitlab/gdk-ee/gitlab/db/fixtures/test/01_plan.rb
/Users/stanhu/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/prometheus-client-mmap-0.7.0.beta18/lib/prometheus/client/helper/mmaped_file.rb:67: [BUG] Segmentation fault at 0x000000010c30e004
ruby 2.3.5p376 (2017-09-14 revision 59905) [x86_64-darwin15]

-- Crash Report log information --------------------------------------------
   See Crash Report log file under the one of following:                    
     * ~/Library/Logs/CrashReporter                                         
     * /Library/Logs/CrashReporter                                          
     * ~/Library/Logs/DiagnosticReports                                     
     * /Library/Logs/DiagnosticReports                                      
   for more details.                                                        
Don't forget to include the above Crash Report log file in bug reports.     

-- Control frame information -----------------------------------------------
c:0099 p:---- s:0419 e:000418 CFUNC  :[]
c:0098 p:---- s:0417 e:000416 CFUNC  :[]
c:0097 p:0010 s:0413 e:000412 METHOD /Users/stanhu/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/prometheus-client-mmap-0.7.0.beta18/lib/prometheus/client/helper/m
c:0096 p:0008 s:0410 e:000409 METHOD /Users/stanhu/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/prometheus-client-mmap-0.7.0.beta18/lib/prometheus/client/helper/m
c:0095 p:0014 s:0402 e:000401 METHOD /Users/stanhu/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/prometheus-client-mmap-0.7.0.beta18/lib/prometheus/client/mmaped_d
c:0094 p:0023 s:0398 e:000397 BLOCK  /Users/stanhu/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/prometheus-client-mmap-0.7.0.beta18/lib/prometheus/client/mmaped_d [FINISH]
c:0093 p:---- s:0396 e:000395 CFUNC  :synchronize
c:0092 p:0011 s:0393 e:000392 METHOD /Users/stanhu/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/prometheus-client-mmap-0.7.0.beta18/lib/prometheus/client/mmaped_d
c:0091 p:0014 s:0387 e:000386 METHOD /Users/stanhu/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/prometheus-client-mmap-0.7.0.beta18/lib/prometheus/client/mmaped_v
c:0090 p:0021 s:0381 e:000380 BLOCK  /Users/stanhu/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/prometheus-client-mmap-0.7.0.beta18/lib/prometheus/client/mmaped_v [FINISH]
c:0089 p:---- s:0379 e:000378 CFUNC  :synchronize
c:0088 p:0011 s:0376 e:000375 METHOD /Users/stanhu/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/prometheus-client-mmap-0.7.0.beta18/lib/prometheus/client/mmaped_v
c:0087 p:0021 s:0372 e:000371 METHOD /Users/stanhu/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/prometheus-client-mmap-0.7.0.beta18/lib/prometheus/client/gauge.rb
c:0086 p:0077 s:0367 e:000366 BLOCK  /Users/stanhu/gitlab/gdk-ee/gitlab/app/services/geo/metrics_update_service.rb:52 [FINISH]
Edited by Stan Hu