Skip to content

Sidekiq not starting after updating Gitlab Omnibus 13.10.10 to 13.12.12-ee: with error "undefined method `prepend_if_ee' for ServiceParams:Module"

Summary

  • Sidekiq not starting after updating Gitlab Omnibus 13.10.10 to 13.12.12-ee

Steps to reproduce

  • Install Gitlab 13.10.10 Omnibus on CentOS 7.9
  • Copy all contents below to another machine with CentOS 8.1
/opt/gitlab
/var/opt/gitlab

  • Set gitlab-runsv service to work, gitlab-ctl reconfigure and check web page is accessible and functional

  • Update Gitlab with

yum -y install gitlab-ee-13.12.12
  • gitlab-ctl reconfigure fails with some error, so delete some file with
rm /opt/gitlab/embedded/service/gitlab-rails/config/feature_flags/development/usage_data_api.yml
rm /opt/gitlab/embedded/service/gitlab-rails/config/initializers/hipchat_client_patch.rb

and gitlab-ctl reconfigure successes

  • try systemctl start gitlab-runsvdir, and runsvdir started normally with
runsvdir -P /opt/gitlab/service log: .......................

  • try to access gitlab web page, shows 502

  • sidekiq fails after gitlab-runsvdir started

What is the current bug behavior?

  • Sidekiq not starts, and gitlab showing 502 error

What is the expected correct behavior?

Sidekiq starts, gitlab-rake gitlab:check success and gitlab not show 502

Relevant logs and/or screenshots

gitlab-rake gitlab:sidekiq:check --trace

** Invoke gitlab:sidekiq:check (first_time)
** Invoke gitlab_environment (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute gitlab_environment
** Execute gitlab:sidekiq:check
Checking Sidekiq ...

Sidekiq: ... Running? ... no
  Try fixing it:
  sudo -u git -H RAILS_ENV=production bin/background_jobs start
  For more information see:
  doc/install/installation.md in section "Install Init Script"
  see log/sidekiq.log for possible errors
  Please fix the error above and rerun the checks.

Checking Sidekiq ... Finished

/var/log/gitlab/sidekiq/current

{"severity":"INFO","time":"2021-11-03T06:05:59.261Z","message":"Cron Jobs - add job with name: geo_sidekiq_cron_config_worker"}
undefined method `prepend_if_ee' for ServiceParams:Module
Did you mean?  prepend
/opt/gitlab/embedded/service/gitlab-rails/app/controllers/concerns/service_params.rb:98:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:324:in `require'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:324:in `block in require'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:291:in `load_dependency'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:324:in `require'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:411:in `block in require_or_load'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:40:in `block in load_interlock'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/dependencies/interlock.rb:14:in `block in loading'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/concurrency/share_lock.rb:151:in `exclusive'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/dependencies/interlock.rb:13:in `loading'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:40:in `load_interlock'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:389:in `require_or_load'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:367:in `depend_on'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:280:in `require_dependency'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.7/lib/rails/engine.rb:481:in `block (2 levels) in eager_load!'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.7/lib/rails/engine.rb:480:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.7/lib/rails/engine.rb:480:in `block in eager_load!'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.7/lib/rails/engine.rb:477:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.7/lib/rails/engine.rb:477:in `eager_load!'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.7/lib/rails/application.rb:509:in `eager_load!'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.7/lib/rails/engine.rb:356:in `eager_load!'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.7/lib/rails/application/finisher.rb:123:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.7/lib/rails/application/finisher.rb:123:in `block in <module:Finisher>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.7/lib/rails/initializable.rb:32:in `instance_exec'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.7/lib/rails/initializable.rb:32:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.7/lib/rails/initializable.rb:61:in `block in run_initializers'
/opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:228:in `block in tsort_each'
/opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
/opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:431:in `each_strongly_connected_component_from'
/opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:349:in `block in each_strongly_connected_component'
/opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:347:in `each'
/opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:347:in `call'
/opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:347:in `each_strongly_connected_component'
/opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:226:in `tsort_each'
/opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:205:in `tsort_each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.7/lib/rails/initializable.rb:60:in `run_initializers'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.7/lib/rails/application.rb:363:in `initialize!'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:324:in `require'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:324:in `block in require'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:291:in `load_dependency'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:324:in `require'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/cli.rb:288:in `boot_system'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/cli.rb:46:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/sidekiq-5.2.9/bin/sidekiq:12:in `<top (required)>'
/opt/gitlab/embedded/bin/sidekiq:23:in `load'
/opt/gitlab/embedded/bin/sidekiq:23:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli/exec.rb:63:in `load'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli/exec.rb:63:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli/exec.rb:28:in `run'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli.rb:476:in `exec'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/vendor/thor/lib/thor.rb:399:in `dispatch'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli.rb:30:in `dispatch'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/vendor/thor/lib/thor/base.rb:476:in `start'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli.rb:24:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.1.4/exe/bundle:46:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/friendly_errors.rb:123:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.1.4/exe/bundle:34:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
{"severity":"INFO","time":"2021-11-03T06:06:10.711Z","message":"A worker terminated, shutting down the cluster"}

/var/log/gitlab/gitlab-shell/gitlab-shell.log

{"correlation_id":"01FKJ7MQYF8Z2C430G0FGK7AZW","duration_ms":3,"error":"Internal API error (502)","level":"error","method":"GET","msg":"Internal API error","status":502,"time":"2021-11-03T06:26:37Z","url":"http://unix/api/v4/internal/check"}

Output of checks

Results of GitLab environment info

gitlab-rake gitlab:env:info

System information
System:
Proxy:          no
Current User:   git
Using RVM:      no
Ruby Version:   2.7.2p137
Gem Version:    3.1.4
Bundler Version:2.1.4
Rake Version:   13.0.3
Redis Version:  6.0.14
Git Version:    2.31.1
Sidekiq Version:5.2.9
Go Version:     unknown

GitLab information
Version:        13.12.12-ee
Revision:       7f1dc4f5c29
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     12.6
URL:            
HTTP Clone URL: 
SSH Clone URL:  g
Elasticsearch:  no
Geo:            no
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers:

GitLab Shell
Version:        13.18.1
Repository storage paths:
- default:      /DATA/BACKUP/gitlab/repositories
- nfs:  /DATA/BACKUP/gitlab/repositories
- cephfs:       /DATA/BACKUP/gitlab/repositories
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell
Git:            /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 13.18.1 ? ... OK (13.18.1)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: FAILED - Internal API error (502)
gitlab-shell self-check failed
  Try fixing it:
  Make sure GitLab is running;
  Check the gitlab-shell configuration file:
  sudo -u git -H editor /opt/gitlab/embedded/service/gitlab-shell/config.yml
  Please fix the error above and rerun the checks.

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK
nfs ... OK
cephfs ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... no
  Try fixing it:
  sudo -u git -H RAILS_ENV=production bin/background_jobs start
  For more information see:
  doc/install/installation.md in section "Install Init Script"
  see log/sidekiq.log for possible errors
  Please fix the error above and rerun the checks.

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab App ...

Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet)
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)

Possible fixes

resolve error in file /opt/gitlab/embedded/service/gitlab-rails/app/controllers/concerns/service_params.rb:98

undefined method `prepend_if_ee' for ServiceParams:Module
Did you mean?  prepend
Edited by Seongwon Kang