Skip to content

Sidekiq queue filling up with JIRA service errors in 8.3

Since upgrading to 8.3 (omnibus edition), JIRA service errors are now filling up the logs, with messages similar to this:

2016-01-05_13:41:24.37468 2016-01-05T13:41:24.374Z 20564 TID-ovmzoffp4 ProjectServiceWorker JID-c2d22c4bf137df2271a46240 INFO: start
2016-01-05_13:41:24.42802 2016-01-05T13:41:24.426Z 20564 TID-ovn587q1k ProjectCacheWorker JID-f7e2d1f8e9a6657965eccd47 INFO: start
2016-01-05_13:41:24.43272 2016-01-05T13:41:24.431Z 20564 TID-ovmzoffp4 ProjectServiceWorker JID-c2d22c4bf137df2271a46240 INFO: fail: 0.058 sec
2016-01-05_13:41:24.43274 2016-01-05T13:41:24.432Z 20564 TID-ovmzoffp4 WARN: {"class"=>"ProjectServiceWorker", "args"=>[817, {"object_kind"=>"push", "before"=>"b7df93708ffd743c87a9eb0d59425f873edf897f", "after"=>"abfc69cd73309d14ecff1d17de4f4adb2b1049d7", "ref"=>"refs/heads/master", "checkout_sha"=>"abfc69cd73309d14ecff1d17de4f4adb2b1049d7", "message"=>nil, "user_id"=>22, "user_name"=>"SCM Build Master", "user_email"=>"svc-engbuildmaster@activevideo.com", "project_id"=>164, "repository"=>{"name"=>"jenkins-master", "url"=>"git@gitlab.example.com:scm/jenkins-master.git", "description"=>"", "homepage"=>"https://gitlab.example.com/scm/jenkins-master", "git_http_url"=>"https://gitlab.example.com/scm/jenkins-master.git", "git_ssh_url"=>"git@gitlab.example.com:scm/jenkins-master.git", "visibility_level"=>0}, "commits"=>[{"id"=>"abfc69cd73309d14ecff1d17de4f4adb2b1049d7", "message"=>"[Jenkins Sync] : SYSTEM: Modification on configuration(s)\n\nChange performed by SYSTEM\n", "timestamp"=>"2016-01-05T14:41:22+01:00", "url"=>"https://gitlab.example.com/scm/jenkins-master/commit/abfc69cd73309d14ecff1d17de4f4adb2b1049d7", "author"=>{"name"=>"svc-engbuildmaster", "email"=>"svc-engbuildmaster@activevideo.com"}, "added"=>[], "modified"=>["jobs/rpm_wrangler/builds/1590/build.xml"], "removed"=>[]}], "total_commits_count"=>1}], "retry"=>true, "queue"=>"project_web_hook", "jid"=>"c2d22c4bf137df2271a46240", "created_at"=>1452001284.3727705, "enqueued_at"=>1452001284.372856, "error_message"=>"undefined method `+' for nil:NilClass", "error_class"=>"NoMethodError", "failed_at"=>1452001284.4313745, "retry_count"=>0}
2016-01-05_13:41:24.43276 2016-01-05T13:41:24.432Z 20564 TID-ovmzoffp4 WARN: NoMethodError: undefined method `+' for nil:NilClass
2016-01-05_13:41:24.43276 2016-01-05T13:41:24.432Z 20564 TID-ovmzoffp4 WARN: /opt/gitlab/embedded/lib/ruby/2.1.0/net/http.rb:1535:in `addr_port'
2016-01-05_13:41:24.43277 /opt/gitlab/embedded/lib/ruby/2.1.0/net/http.rb:1468:in `begin_transport'
2016-01-05_13:41:24.43277 /opt/gitlab/embedded/lib/ruby/2.1.0/net/http.rb:1410:in `transport_request'
2016-01-05_13:41:24.43277 /opt/gitlab/embedded/lib/ruby/2.1.0/net/http.rb:1384:in `request'
2016-01-05_13:41:24.43277 /opt/gitlab/embedded/lib/ruby/2.1.0/net/http.rb:1377:in `block in request'
2016-01-05_13:41:24.43277 /opt/gitlab/embedded/lib/ruby/2.1.0/net/http.rb:853:in `start'
2016-01-05_13:41:24.43278 /opt/gitlab/embedded/lib/ruby/2.1.0/net/http.rb:1375:in `request'
2016-01-05_13:41:24.43278 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/httparty-0.13.7/lib/httparty/request.rb:117:in `perform'
2016-01-05_13:41:24.43278 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/httparty-0.13.7/lib/httparty.rb:545:in `perform_request'
2016-01-05_13:41:24.43278 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/httparty-0.13.7/lib/httparty.rb:476:in `get'
2016-01-05_13:41:24.43278 /opt/gitlab/embedded/service/gitlab-rails/app/models/project_services/jira_service.rb:123:in `test_settings'
2016-01-05_13:41:24.43278 /opt/gitlab/embedded/service/gitlab-rails/app/models/project_services/jira_service.rb:86:in `execute'
2016-01-05_13:41:24.43279 /opt/gitlab/embedded/service/gitlab-rails/app/workers/project_service_worker.rb:8:in `perform'
2016-01-05_13:41:24.43279 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:150:in `execute_job'
2016-01-05_13:41:24.43279 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:132:in `block (2 levels) in process'
2016-01-05_13:41:24.43279 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:127:in `block in invoke'
2016-01-05_13:41:24.43279 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/memory_killer.rb:17:in `call'
2016-01-05_13:41:24.43280 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
2016-01-05_13:41:24.43280 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/server/active_record.rb:6:in `call'
2016-01-05_13:41:24.43280 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
2016-01-05_13:41:24.43280 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/server/retry_jobs.rb:74:in `call'
2016-01-05_13:41:24.43281 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
2016-01-05_13:41:24.43281 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/server/logging.rb:11:in `block in call'
2016-01-05_13:41:24.43281 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/logging.rb:30:in `with_context'
2016-01-05_13:41:24.43281 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/server/logging.rb:7:in `call'
2016-01-05_13:41:24.43282 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
2016-01-05_13:41:24.43282 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:132:in `call'
2016-01-05_13:41:24.43282 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:132:in `invoke'
2016-01-05_13:41:24.43282 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:127:in `block in process'
2016-01-05_13:41:24.43282 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:166:in `stats'
2016-01-05_13:41:24.43282 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:126:in `process'
2016-01-05_13:41:24.43283 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:79:in `process_one'
2016-01-05_13:41:24.43283 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:67:in `run'
2016-01-05_13:41:24.43283 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/util.rb:16:in `watchdog'
2016-01-05_13:41:24.43283 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/util.rb:24:in `block in safe_thread'
2016-01-05_13:41:24.48809 2016-01-05T13:41:24.487Z 20564 TID-u0wks PostReceive JID- INFO: done: 0.902 sec

Of course this has always been the community edition, so it used the 'old' style of JIRA integration, which apparently has been auto-upgraded when going from 8.2 to 8.3.

At the moment I do not have (nor can I personally create) a special GitLab user in JIRA. This will have to go through a long and tedious procedure, so any quick workaround would be appreciated!