Seg fault in inspecting heap dump
In https://nonprod-log.gitlab.net/goto/41999c50-c42a-11ed-9af2-6131f0ee4ce6, I saw a seg fault in staging with this partial backtrace:
/usr/lib/ruby/3.0.0/objspace.rb:87: [BUG] Segmentation fault at 0x00007f3d94e35000
ruby 3.0.5p211 (2022-11-24 revision ba5cf0f7c5) [x86_64-linux]
-- Control frame information -----------------------------------------------
c:0054 p:---- s:0324 e:000323 CFUNC :_dump_all
c:0053 p:0130 s:0317 e:000316 METHOD /usr/lib/ruby/3.0.0/objspace.rb:87
c:0052 p:0023 s:0307 e:000306 METHOD /srv/gitlab/lib/gitlab/memory/reports/heap_dump.rb:28
c:0051 p:0068 s:0302 e:000301 BLOCK /srv/gitlab/lib/gitlab/memory/reporter.rb:94 [FINISH]
c:0050 p:---- s:0296 e:000295 CFUNC :open
c:0049 p:0058 s:0290 e:000289 METHOD /srv/gitlab/lib/gitlab/memory/reporter.rb:84
c:0048 p:0063 s:0277 e:000276 METHOD /srv/gitlab/lib/gitlab/memory/reporter.rb:72
c:0047 p:0071 s:0269 e:000268 METHOD /srv/gitlab/lib/gitlab/memory/reporter.rb:32
c:0046 p:0045 s:0258 e:000257 BLOCK /srv/gitlab/config/initializers/diagnostic_reports.rb:12
c:0045 p:0005 s:0255 e:000254 BLOCK /srv/gitlab/lib/gitlab/cluster/lifecycle_events.rb:173 [FINISH]
c:0044 p:---- s:0250 e:000249 CFUNC :each
c:0043 p:0011 s:0246 e:000245 METHOD /srv/gitlab/lib/gitlab/cluster/lifecycle_events.rb:172
c:0042 p:0008 s:0240 e:000239 METHOD /srv/gitlab/lib/gitlab/cluster/lifecycle_events.rb:154
c:0041 p:0020 s:0236 e:000235 BLOCK /srv/gitlab/config/puma.rb:96
c:0040 p:0007 s:0233 e:000232 BLOCK /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/puma-5.6.5/lib/puma/configuration.rb:297 [FINISH]
c:0039 p:---- s:0228 e:000227 CFUNC :each
c:0038 p:0010 s:0224 e:000223 METHOD /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/puma-5.6.5/lib/puma/configuration.rb:295
c:0037 p:0493 s:0217 E:000d20 METHOD /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/puma-5.6.5/lib/puma/cluster/worker.rb:141
c:0036 p:0104 s:0205 e:000204 METHOD /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/puma-5.6.5/lib/puma/cluster.rb:204
c:0035 p:0008 s:0196 e:000195 BLOCK /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/puma-5.6.5/lib/puma/cluster.rb:97
c:0034 p:0015 s:0193 e:000192 BLOCK /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/fork_tracker.rb:10
c:0033 p:0015 s:0190 e:000189 BLOCK /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/fork_tracker.rb:10 [FINISH]
c:0032 p:---- s:0187 e:000186 CFUNC :fork
c:0031 p:0013 s:0183 e:000182 METHOD /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/fork_tracker.rb:8
c:0030 p:0008 s:0177 e:000176 METHOD /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/fork_tracker.rb:27
c:0029 p:0013 s:0172 e:000171 METHOD /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/fork_tracker.rb:8
c:0028 p:0008 s:0166 e:000165 METHOD /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/fork_tracker.rb:27
c:0027 p:0021 s:0161 e:000160 METHOD /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/puma-5.6.5/lib/puma/cluster.rb:97
c:0026 p:0056 s:0154 e:000153 BLOCK /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/puma-5.6.5/lib/puma/cluster.rb:79 [FINISH]
c:0025 p:---- s:0149 e:000148 CFUNC :times
c:0024 p:0060 s:0145 e:000144 METHOD /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/puma-5.6.5/lib/puma/cluster.rb:72
c:0023 p:0058 s:0139 e:000138 METHOD /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/puma-5.6.5/lib/puma/cluster.rb:161
c:0022 p:0479 s:0134 E:000cf8 METHOD /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/puma-5.6.5/lib/puma/cluster.rb:435
c:0021 p:0114 s:0117 e:000116 METHOD /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/puma-5.6.5/lib/puma/launcher.rb:193
c:0020 p:0005 s:0110 e:000109 METHOD /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/puma-5.6.5/lib/puma/cli.rb:81
c:0019 p:0036 s:0106 e:000105 TOP /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/puma-5.6.5/bin/puma:10 [FINISH]
c:0018 p:---- s:0102 e:000101 CFUNC :load
c:0017 p:0124 s:0097 e:000096 TOP /srv/gitlab/vendor/bundle/ruby/3.0.0/bin/puma:25 [FINISH]
c:0016 p:---- s:0092 e:000091 CFUNC :load
c:0015 p:0107 s:0087 e:000086 METHOD /usr/lib/ruby/site_ruby/3.0.0/bundler/cli/exec.rb:58
c:0014 p:0071 s:0081 e:000080 METHOD /usr/lib/ruby/site_ruby/3.0.0/bundler/cli/exec.rb:23
c:0013 p:0055 s:0076 e:000075 METHOD /usr/lib/ruby/site_ruby/3.0.0/bundler/cli.rb:492
c:0012 p:0054 s:0071 e:000070 METHOD /usr/lib/ruby/site_ruby/3.0.0/bundler/vendor/thor/lib/thor/command.rb:27
c:0011 p:0040 s:0063 e:000062 METHOD /usr/lib/ruby/site_ruby/3.0.0/bundler/vendor/thor/lib/thor/invocation.rb:127
c:0010 p:0239 s:0056 e:000055 METHOD /usr/lib/ruby/site_ruby/3.0.0/bundler/vendor/thor/lib/thor.rb:392
c:0009 p:0008 s:0043 e:000042 METHOD /usr/lib/ruby/site_ruby/3.0.0/bundler/cli.rb:34
c:0008 p:0066 s:0038 e:000037 METHOD /usr/lib/ruby/site_ruby/3.0.0/bundler/vendor/thor/lib/thor/base.rb:485
c:0007 p:0036 s:0031 e:000030 METHOD /usr/lib/ruby/site_ruby/3.0.0/bundler/cli.rb:28
c:0006 p:0109 s:0026 e:000025 BLOCK /usr/lib/ruby/gems/3.0.0/gems/bundler-2.4.8/exe/bundle:45
c:0005 p:0014 s:0020 e:000019 METHOD /usr/lib/ruby/site_ruby/3.0.0/bundler/friendly_errors.rb:117
c:0004 p:0227 s:0015 E:0011d8 TOP /usr/lib/ruby/gems/3.0.0/gems/bundler-2.4.8/exe/bundle:33 [FINISH]
c:0003 p:---- s:0011 e:000010 CFUNC :load
c:0002 p:0024 s:0006 e:000005 EVAL /srv/gitlab/bin/bundle:5 [FINISH]
c:0001 p:0000 s:0003 E:001410 (none) [FINISH]
-- Ruby level backtrace information ----------------------------------------
/srv/gitlab/bin/bundle:5:in `<main>'
/srv/gitlab/bin/bundle:5:in `load'
/usr/lib/ruby/gems/3.0.0/gems/bundler-2.4.8/exe/bundle:33:in `<top (required)>'
/usr/lib/ruby/site_ruby/3.0.0/bundler/friendly_errors.rb:117:in `with_friendly_errors'
/usr/lib/ruby/gems/3.0.0/gems/bundler-2.4.8/exe/bundle:45:in `block in <top (required)>'
/usr/lib/ruby/site_ruby/3.0.0/bundler/cli.rb:28:in `start'
/usr/lib/ruby/site_ruby/3.0.0/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
/usr/lib/ruby/site_ruby/3.0.0/bundler/cli.rb:34:in `dispatch'
/usr/lib/ruby/site_ruby/3.0.0/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
/usr/lib/ruby/site_ruby/3.0.0/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/usr/lib/ruby/site_ruby/3.0.0/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/usr/lib/ruby/site_ruby/3.0.0/bundler/cli.rb:492:in `exec'
/usr/lib/ruby/site_ruby/3.0.0/bundler/cli/exec.rb:23:in `run'
/usr/lib/ruby/site_ruby/3.0.0/bundler/cli/exec.rb:58:in `kernel_load'
/usr/lib/ruby/site_ruby/3.0.0/bundler/cli/exec.rb:58:in `load'
/srv/gitlab/vendor/bundle/ruby/3.0.0/bin/puma:25:in `<top (required)>'
/srv/gitlab/vendor/bundle/ruby/3.0.0/bin/puma:25:in `load'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/puma-5.6.5/bin/puma:10:in `<top (required)>'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/puma-5.6.5/lib/puma/cli.rb:81:in `run'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/puma-5.6.5/lib/puma/launcher.rb:193:in `run'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/puma-5.6.5/lib/puma/cluster.rb:435:in `run'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/puma-5.6.5/lib/puma/cluster.rb:161:in `check_workers'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/puma-5.6.5/lib/puma/cluster.rb:72:in `spawn_workers'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/puma-5.6.5/lib/puma/cluster.rb:72:in `times'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/puma-5.6.5/lib/puma/cluster.rb:79:in `block in spawn_workers'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/puma-5.6.5/lib/puma/cluster.rb:97:in `spawn_worker'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/fork_tracker.rb:27:in `fork'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/fork_tracker.rb:8:in `fork'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/fork_tracker.rb:27:in `fork'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/fork_tracker.rb:8:in `fork'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/fork_tracker.rb:8:in `fork'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/fork_tracker.rb:10:in `block in fork'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/fork_tracker.rb:10:in `block in fork'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/puma-5.6.5/lib/puma/cluster.rb:97:in `block in spawn_worker'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/puma-5.6.5/lib/puma/cluster.rb:204:in `worker'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/puma-5.6.5/lib/puma/cluster/worker.rb:141:in `run'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/puma-5.6.5/lib/puma/configuration.rb:295:in `run_hooks'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/puma-5.6.5/lib/puma/configuration.rb:295:in `each'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/puma-5.6.5/lib/puma/configuration.rb:297:in `block in run_hooks'
/srv/gitlab/config/puma.rb:96:in `block in _load_from'
/srv/gitlab/lib/gitlab/cluster/lifecycle_events.rb:154:in `do_worker_stop'
/srv/gitlab/lib/gitlab/cluster/lifecycle_events.rb:172:in `call'
/srv/gitlab/lib/gitlab/cluster/lifecycle_events.rb:172:in `each'
/srv/gitlab/lib/gitlab/cluster/lifecycle_events.rb:173:in `block in call'
/srv/gitlab/config/initializers/diagnostic_reports.rb:12:in `block in <main>'
/srv/gitlab/lib/gitlab/memory/reporter.rb:32:in `run_report'
/srv/gitlab/lib/gitlab/memory/reporter.rb:72:in `store_report'
/srv/gitlab/lib/gitlab/memory/reporter.rb:84:in `write_compressed_file'
/srv/gitlab/lib/gitlab/memory/reporter.rb:84:in `open'
/srv/gitlab/lib/gitlab/memory/reporter.rb:94:in `block in write_compressed_file'
/srv/gitlab/lib/gitlab/memory/reports/heap_dump.rb:28:in `run'
/usr/lib/ruby/3.0.0/objspace.rb:87:in `dump_all'
/usr/lib/ruby/3.0.0/objspace.rb:87:in `_dump_all'
-- Machine register context ------------------------------------------------
RIP: 0x00007f3dcdcd54c0 RBP: 0x00007ffc4aa13640 RSP: 0x00007ffc4aa13600
RAX: 0x8080808080808080 RBX: 0x00007f3dcc7d0000 RCX: 0x00007f3d94e3cff9
RDX: 0x8080808080808080 RDI: 0x00007f3dcc7d0000 RSI: 0x0000000000008000
R8: 0x0000000000000000 R9: 0x00007f3d9a296900 R10: 0x0000000000000002
R11: 0x0000000000000008 R12: 0x00007f3d94e35000 R13: 0x0000000000008000
R14: 0x00007f3d94e3d000 R15: 0x00007ffc4aa138c0 EFL: 0x0000000000010293
-- C level backtrace information -------------------------------------------
/usr/lib/libruby.so.3.0(rb_print_backtrace+0x11) [0x7f3dcdd6a57e] vm_dump.This is probably #39 (closed) happening again, but I created a separate issue since this also appears to be failing occasionally in CI now, at least with Ruby 3.2: https://gitlab.com/gitlab-org/ruby/gems/prometheus-client-mmap/-/jobs/3934996179