Sidekiq segfaults on arm64

The following discussion from !4535 (merged) should be addressed:

  • @twk3 started a discussion: (+4 comments)

    I'm getting ruby segfaults in both sidekiq and puma:

    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/method_source-1.0.0/lib/method_source/code_helpers.rb:71: [BUG] Segmentation fault at 0x00000000000000dc
    ruby 2.6.6p146 (2020-03-31 revision 67876) [aarch64-linux]
    
    -- Control frame information -----------------------------------------------
    c:0063 p:---- s:0355 e:000354 CFUNC  :eval
    c:0062 p:0021 s:0350 e:000349 BLOCK  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/method_source-1.0.0/lib/method_source/code_helpers.rb:71 [FINISH]
    c:0061 p:---- s:0347 e:000346 CFUNC  :catch
    c:0060 p:0014 s:0342 e:000341 METHOD /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/method_source-1.0.0/lib/method_source/code_helpers.rb:70
    c:0059 p:0030 s:0336 e:000335 BLOCK  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/method_source-1.0.0/lib/method_source/code_helpers.rb:97 [FINISH]
    c:0058 p:---- s:0332 e:000331 CFUNC  :each
    c:0057 p:0039 s:0328 e:000327 METHOD /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/method_source-1.0.0/lib/method_source/code_helpers.rb:95
    c:0056 p:0077 s:0320 e:000319 METHOD /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/method_source-1.0.0/lib/method_source/code_helpers.rb:30
    c:0055 p:0060 s:0310 e:000309 METHOD /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/method_source-1.0.0/lib/method_source.rb:27
    c:0054 p:0031 s:0301 e:000300 METHOD /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/method_source-1.0.0/lib/method_source.rb:110
    c:0053 p:0051 s:0297 e:000296 BLOCK  /opt/gitlab/embedded/service/gitlab-rails/config/initializers/zz_metrics.rb:181
    c:0052 p:0046 s:0291 e:000290 BLOCK  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:78 [FINISH]
    c:0051 p:---- s:0286 e:000285 CFUNC  :each
    c:0050 p:0025 s:0282 e:000281 METHOD /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:74
    c:0049 p:0030 s:0276 e:000275 METHOD /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:57
    c:0048 p:0093 s:0268 e:000267 BLOCK  /opt/gitlab/embedded/service/gitlab-rails/config/initializers/zz_metrics.rb:172
    c:0047 p:0017 s:0263 e:000262 BLOCK  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/lazy_load_hooks.rb:68
    c:0046 p:0037 s:0260 e:000259 METHOD /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/lazy_load_hooks.rb:61
    c:0045 p:0016 s:0253 e:000252 METHOD /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/lazy_load_hooks.rb:66
    c:0044 p:0013 s:0245 e:000244 BLOCK  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/lazy_load_hooks.rb:52 [FINISH]
    c:0043 p:---- s:0240 e:000239 CFUNC  :each
    c:0042 p:0035 s:0236 e:000235 METHOD /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/lazy_load_hooks.rb:51
    c:0041 p:0014 s:0230 e:000229 BLOCK  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.3.1/lib/rails/application/finisher.rb:129 [FINISH]
    c:0040 p:---- s:0227 e:000226 CFUNC  :instance_exec
    c:0039 p:0015 s:0222 e:000221 METHOD /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.3.1/lib/rails/initializable.rb:32
    c:0038 p:0019 s:0217 e:000216 BLOCK  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.3.1/lib/rails/initializable.rb:61
    c:0037 p:0019 s:0213 e:000212 BLOCK  /opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:228
    c:0036 p:0005 s:0209 e:000208 BLOCK  /opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:350
    c:0035 p:0111 s:0205 E:001c60 METHOD /opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:431
    c:0034 p:0031 s:0193 E:000ef8 BLOCK  /opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:349 [FINISH]
    c:0033 p:---- s:0189 e:000188 CFUNC  :each
    c:0032 p:---- s:0186 e:000185 CFUNC  :call
    c:0031 p:0033 s:0182 E:0017b0 METHOD /opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:347
    c:0030 p:0034 s:0174 E:001d40 METHOD /opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:226
    c:0029 p:0034 s:0168 e:000167 METHOD /opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:205
    c:0028 p:0022 s:0161 E:001ce0 METHOD /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.3.1/lib/rails/initializable.rb:60
    c:0027 p:0023 s:0155 e:000154 METHOD /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.3.1/lib/rails/application.rb:363
    c:0026 p:0020 s:0150 e:000149 TOP    /opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:5 [FINISH]
    c:0025 p:---- s:0147 e:000146 CFUNC  :require
    c:0024 p:0008 s:0142 e:000141 BLOCK  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/dependencies.rb:324
    c:0023 p:0068 s:0139 e:000138 METHOD /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/dependencies.rb:291
    c:0022 p:0011 s:0132 e:000131 METHOD /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/dependencies.rb:324
    c:0021 p:0267 s:0126 e:000125 METHOD /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/cli.rb:288
    c:0020 p:0025 s:0122 e:000121 METHOD /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/cli.rb:46
    c:0019 p:0036 s:0112 e:000111 TOP    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.9/bin/sidekiq:12 [FINISH]
    c:0018 p:---- s:0107 e:000106 CFUNC  :load
    c:0017 p:0132 s:0102 e:000101 TOP    /opt/gitlab/embedded/bin/sidekiq:23 [FINISH]
    c:0016 p:---- s:0097 e:000096 CFUNC  :load
    c:0015 p:0129 s:0092 e:000091 METHOD /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74
    c:0014 p:0075 s:0083 e:000082 METHOD /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:28
    c:0013 p:0026 s:0078 e:000077 METHOD /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:463
    c:0012 p:0064 s:0073 e:000072 METHOD /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/command.rb:27
    c:0011 p:0047 s:0065 e:000064 METHOD /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/invocation.rb:126
    c:0010 p:0261 s:0058 e:000057 METHOD /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor.rb:387
    c:0009 p:0009 s:0045 e:000044 METHOD /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:27
    c:0008 p:0064 s:0040 e:000039 METHOD /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/base.rb:466
    c:0007 p:0009 s:0033 e:000032 METHOD /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:18
    c:0006 p:0076 s:0027 e:000026 BLOCK  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:30
    c:0005 p:0002 s:0021 e:000020 METHOD /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/friendly_errors.rb:124
    c:0004 p:0048 s:0016 E:001e10 TOP    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:22 [FINISH]
    c:0003 p:---- s:0013 e:000012 CFUNC  :load
    c:0002 p:0132 s:0008 E:001150 EVAL   /opt/gitlab/embedded/bin/bundle:23 [FINISH]
    c:0001 p:0000 s:0003 E:000b50 (none) [FINISH]
    
    -- Ruby level backtrace information ----------------------------------------
    /opt/gitlab/embedded/bin/bundle:23:in `<main>'
    /opt/gitlab/embedded/bin/bundle:23:in `load'
    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:22:in `<top (required)>'
    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:30:in `block in <top (required)>'
    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:18:in `start'
    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:27:in `dispatch'
    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:463:in `exec'
    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:28:in `run'
    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `kernel_load'
    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `load'
    /opt/gitlab/embedded/bin/sidekiq:23:in `<top (required)>'
    /opt/gitlab/embedded/bin/sidekiq:23:in `load'
    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.9/bin/sidekiq:12:in `<top (required)>'
    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/cli.rb:46:in `run'
    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/cli.rb:288:in `boot_system'
    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/dependencies.rb:324:in `require'
    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/dependencies.rb:291:in `load_dependency'
    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/dependencies.rb:324:in `block in require'
    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/dependencies.rb:324:in `require'
    /opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:5:in `<top (required)>'
    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.3.1/lib/rails/application.rb:363:in `initialize!'
    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.3.1/lib/rails/initializable.rb:60:in `run_initializers'
    /opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:205:in `tsort_each'
    /opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:226:in `tsort_each'
    /opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:347:in `each_strongly_connected_component'
    /opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:347:in `call'
    /opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:347:in `each'
    /opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:349:in `block in each_strongly_connected_component'
    /opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:431:in `each_strongly_connected_component_from'
    /opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
    /opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:228:in `block in tsort_each'
    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.3.1/lib/rails/initializable.rb:61:in `block in run_initializers'
    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.3.1/lib/rails/initializable.rb:32:in `run'
    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.3.1/lib/rails/initializable.rb:32:in `instance_exec'
    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.3.1/lib/rails/application/finisher.rb:129:in `block in <module:Finisher>'
    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/lazy_load_hooks.rb:51:in `run_load_hooks'
    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/lazy_load_hooks.rb:51:in `each'
    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/lazy_load_hooks.rb:52:in `block in run_load_hooks'
    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/lazy_load_hooks.rb:66:in `execute_hook'
    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/lazy_load_hooks.rb:61:in `with_execution_control'
    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/lazy_load_hooks.rb:68:in `block in execute_hook'
    /opt/gitlab/embedded/service/gitlab-rails/config/initializers/zz_metrics.rb:172:in `block in <top (required)>'
    /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:57:in `instrument_class_hierarchy'
    /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:74:in `instrument_methods'
    /opt/gitlab/embedded/service/gitlab-rails/config/initializers/zz_metrics.rb:181:in `block (2 levels) in <top (required)>'
    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/method_source-1.0.0/lib/method_source.rb:110:in `source'
    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/method_source-1.0.0/lib/method_source.rb:27:in `source_helper'
    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/method_source-1.0.0/lib/method_source/code_helpers.rb:30:in `expression_at'
    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/method_source-1.0.0/lib/method_source/code_helpers.rb:95:in `extract_first_expression'
    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/method_source-1.0.0/lib/method_source/code_helpers.rb:95:in `each'
    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/method_source-1.0.0/lib/method_source/code_helpers.rb:97:in `block in extract_first_expression'
    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/method_source-1.0.0/lib/method_source/code_helpers.rb:70:in `complete_expression?'
    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/method_source-1.0.0/lib/method_source/code_helpers.rb:70:in `catch'
    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/method_source-1.0.0/lib/method_source/code_helpers.rb:71:in `block in complete_expression?'
    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/method_source-1.0.0/lib/method_source/code_helpers.rb:71:in `eval'

    This is on an aws a1.xlarge running the openSUSE Leap 15.1 ARM marketplace ami image.