GitLab is not responding after upgrade to 12.8
Summary
After upgrading a GitLab instance to 12.8, GitLab doesn't come back up again.
This problem seems to affect GitLab instances that have a license installed that is close to expiring.
Originally reported by ZD and ZD (internal use only)
Steps to reproduce
- Have a working GitLab instance on
12.7.6
, and load a Ultimate license into GitLab that will expire in the next 3 days. - Upgrade GitLab to
12.8.0
via the package manager of the OS - The initial upgrade will display an error on the migration stage -
undefined method `pluralize' for LicenseHelper:Module
but the upgrade seems to continue running. If you access the GitLab UI afterwards you will see a 502 error. - The logs will contain
undefined method `pluralize' for LicenseHelper:Module
when you attempt to view the GitLab UI - see Relevant logs and/or screenshots for stacktrace
What is the current bug behavior?
After an upgrade to GitLab 12.8, if the license is nearing expiry, the GitLab instance becomes unavailable.
What is the expected correct behavior?
After an upgrade to GitLab 12.8, if the license is nearing expiry, GitLab should continue to operate normally.
Relevant logs and/or screenshots
undefined method `pluralize' for LicenseHelper:Module
/opt/gitlab/embedded/service/gitlab-rails/ee/app/helpers/license_helper.rb:60:in `expiration_message'
/opt/gitlab/embedded/service/gitlab-rails/ee/app/helpers/license_helper.rb:29:in `license_message'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/0_license.rb:14:in `block in <top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab.rb:95:in `ee'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/0_license.rb:3:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/dependencies.rb:319:in `load'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/dependencies.rb:319:in `block in load'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/dependencies.rb:291:in `load_dependency'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/dependencies.rb:319:in `load'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/engine.rb:667:in `block in load_config_initializer'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/notifications.rb:182:in `instrument'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/engine.rb:666:in `load_config_initializer'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/engine.rb:624:in `block (2 levels) in <class:Engine>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/engine.rb:623:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/engine.rb:623:in `block in <class:Engine>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/initializable.rb:32:in `instance_exec'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/initializable.rb:32:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/initializable.rb:61:in `block in run_initializers'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:228:in `block in tsort_each'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:422:in `block (2 levels) in each_strongly_connected_component_from'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:431:in `each_strongly_connected_component_from'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:421:in `block in each_strongly_connected_component_from'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/initializable.rb:50:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/initializable.rb:50:in `tsort_each_child'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:415:in `call'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:415:in `each_strongly_connected_component_from'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:349:in `block in each_strongly_connected_component'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:347:in `each'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:347:in `call'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:347:in `each_strongly_connected_component'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:226:in `tsort_each'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:205:in `tsort_each'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/initializable.rb:60:in `run_initializers'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/application.rb:363:in `initialize!'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:5:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/dependencies.rb:325:in `require'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/dependencies.rb:325:in `block in require'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/dependencies.rb:291:in `load_dependency'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/dependencies.rb:325:in `require'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:288:in `boot_system'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:46:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/bin/sidekiq:12:in `<top (required)>'
/opt/gitlab/embedded/bin/sidekiq:23:in `load'
/opt/gitlab/embedded/bin/sidekiq:23:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `load'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:28:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:463:in `exec'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:27:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:18:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:30:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:22:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Output of checks
(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)
Results of GitLab environment info
Expand for output related to GitLab environment info
sudo gitlab-rake gitlab:env:infoSystem information System: Ubuntu 16.04 Proxy: no Current User: git Using RVM: no Ruby Version: 2.6.5p114 Gem Version: 2.7.10 Bundler Version:1.17.3 Rake Version: 12.3.3 Redis Version: 5.0.7 Git Version: 2.24.1 Sidekiq Version:5.2.7 Go Version: unknown
GitLab information Version: 12.8.0-ee Revision: 0bd32f78864 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 10.12 URL: http://gitlab.example.com HTTP Clone URL: http://gitlab.example.com/some-group/some-project.git SSH Clone URL: git@gitlab.example.com:some-group/some-project.git Elasticsearch: no Geo: no Using LDAP: no Using Omniauth: yes Omniauth Providers:
GitLab Shell Version: 11.0.0 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
sudo gitlab-rake gitlab:check SANITIZE=true Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 11.0.0 ? ... OK (11.0.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet) Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... can't check, you have no projects Redis version >= 2.8.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.5) Git version >= 2.22.0 ? ... yes (2.24.1) Git user has default SSH configuration? ... yes Active users: ... 1 Is authorized keys file accessible? ... yes Elasticsearch version 5.6 - 6.x? ... skipped (elasticsearch is disabled)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
Possible fixes
- Change the contents of
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/0_license.rb
to match https://gitlab.com/gitlab-org/gitlab/-/blob/2b3d00a77822eaf2e622dd0b1baf85ebea2b1ee4/config/initializers/0_license.rb - Run
sudo gitlab-ctl reconfigure
- GitLab should now pop up