After upgrading to 8.4, sidekiq start continuously crashing
For some reason @push_commits
is nil
in /home/git/gitlab/app/services/git_push_service.rb, when calling GitPushService.build_push_data
. This makes Sidekiq eating CPU very intensively (up to 100%), showing "sidekiq 4.0.1 gitlab [20 of 25 busy]" in ps
output.
Here is call stack:
2016-01-24T03:27:40.899Z 18527 TID-ovs44fjxc PostReceive JID- INFO: fail: 181.219 sec
2016-01-24T03:27:40.899Z 18527 TID-ovs44fjxc WARN: {"class"=>"PostReceive", "args"=>["/home/git/repositories/mirror/vim-fugitive.git", "key-54", "NWM3MjNkNDhiMGZjYmRjNzdkYjdiODU3Y2Y4ZThjZjA2NzNhOTliZiBmZmU0\nYzQzNjM4YzQ4MGNjMmNiMzIzNDc4YzA4YmU1NTRmMTkzN2U4IHJlZnMvcHVs\nbC83MTQvbWVyZ2UK\n"]}
2016-01-24T03:27:40.899Z 18527 TID-ovs44fjxc WARN: NoMethodError: undefined method `size' for nil:NilClass
2016-01-24T03:27:40.899Z 18527 TID-ovs44fjxc WARN: /home/git/gitlab/lib/gitlab/push_data_builder.rb:26:in `build'
/home/git/gitlab/app/services/git_push_service.rb:104:in `build_push_data'
/home/git/gitlab/app/services/git_push_service.rb:59:in `execute'
/home/git/gitlab/app/workers/post_receive.rb:41:in `block in perform'
/home/git/gitlab/app/workers/post_receive.rb:28:in `each'
/home/git/gitlab/app/workers/post_receive.rb:28:in `perform'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:150:in `execute_job'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:132:in `block (2 levels) in process'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:127:in `block in invoke'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/server/active_record.rb:6:in `call'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/server/retry_jobs.rb:74:in `call'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/server/logging.rb:11:in `block in call'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/logging.rb:30:in `with_context'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/server/logging.rb:7:in `call'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:132:in `call'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:132:in `invoke'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:127:in `block in process'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:166:in `stats'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:126:in `process'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:79:in `process_one'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:67:in `run'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/util.rb:16:in `watchdog'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/util.rb:24:in `block in safe_thread'
2016-01-24T03:45:02.859Z 18527 TID-ovt6cgbfg PostReceive JID- INFO: fail: 9406.519 sec