Update fails during gitlab-ctl reconfigure: undefined method + for nil
Summary
Update from 11.4.5-ee to 11.8.1-ee fails with
undefined method `+' for nil:NilClassduring gitlab-ctl reconfigure
Steps to reproduce
sudo apt-get update
sudo apt-get install gitlab-ee Example Project
(Not specific to any project).
What is the current bug behavior?
sudo apt-get install gitlab-ee fails with the error undefined method +' for nil:NilClass`
What is the expected correct behavior?
The update should finish without the above error.
Relevant logs and/or screenshots
Console log:
root@zakharovite:/mnt/gitlab/backups.new# sudo apt-get install gitlab-ee
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
aufs-tools augeas-lenses cgroupfs-mount javascript-common libaugeas0 libdrm2 libjs-jquery libnih-dbus1 libnih1 libruby2.1 libyaml-0-2 mountall pigz plymouth ruby ruby-augeas ruby-hiera ruby-json ruby-rgen ruby-safe-yaml ruby-selinux
ruby-shadow ruby2.1 rubygems-integration virt-what
Use 'apt-get autoremove' to remove them.
The following packages will be upgraded:
gitlab-ee
1 upgraded, 0 newly installed, 0 to remove and 197 not upgraded.
Need to get 634 MB of archives.
After this operation, 247 MB of additional disk space will be used.
Get:1 https://packages.gitlab.com/gitlab/gitlab-ee/debian/ jessie/main gitlab-ee amd64 11.9.0-ee.0 [634 MB]
Fetched 634 MB in 30s (20.9 MB/s)
Reading changelogs... Done
(Reading database ... 144171 files and directories currently installed.)
Preparing to unpack .../gitlab-ee_11.9.0-ee.0_amd64.deb ...
gitlab preinstall: Automatically backing up only the GitLab SQL database (excluding everything else!)
Dumping database ...
Dumping PostgreSQL database gitlabhq_production ... [DONE]
done
Dumping repositories ...
[SKIPPED]
Dumping uploads ...
[SKIPPED]
Dumping builds ...
[SKIPPED]
Dumping artifacts ...
[SKIPPED]
Dumping pages ...
[SKIPPED]
Dumping lfs objects ...
[SKIPPED]
Dumping container registry images ...
[SKIPPED]
Creating backup archive: 1553528604_2019_03_25_11.4.5-ee_gitlab_backup.tar ... done
Uploading backup archive to remote storage ... skipped
Deleting tmp directories ... done
done
Deleting old backups ... skipping
Unpacking gitlab-ee (11.9.0-ee.0) over (11.4.5-ee.0) ...
Setting up gitlab-ee (11.9.0-ee.0) ...
Checking PostgreSQL executables:Starting Chef Client, version 13.6.4
resolving cookbooks for run list: ["gitlab::config", "postgresql::bin"]
Synchronizing Cookbooks:
- redis (0.1.0)
- registry (0.1.0)
- mattermost (0.1.0)
- gitaly (0.1.0)
- letsencrypt (0.1.0)
- nginx (0.1.0)
- postgresql (0.1.0)
- package (0.1.0)
- gitlab (0.0.1)
- crond (0.1.0)
- consul (0.1.0)
- runit (4.3.0)
- acme (3.1.0)
- compat_resource (12.19.1)
Installing Cookbook Gems:
Compiling Cookbooks...
================================================================================
Recipe Compile Error in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/config.rb
================================================================================
NoMethodError
-------------
undefined method `+' for nil:NilClass
Cookbook Trace:
---------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/gitlab_pages.rb:84:in `parse_gitlab_pages_daemon'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/gitlab_pages.rb:28:in `parse_variables'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/settings_helper.rb:174:in `block in generate_config'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/settings_helper.rb:172:in `each'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/settings_helper.rb:172:in `generate_config'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/config.rb:26:in `from_file'
Relevant File Content:
----------------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/gitlab_pages.rb:
77: Gitlab['gitlab_pages']['cert_key'] ||= "/etc/gitlab/ssl/#{Gitlab['gitlab_pages']['domain']}.key"
78: end
79:
80: Gitlab['gitlab_pages']['pages_root'] ||= (Gitlab['gitlab_rails']['pages_path'] || File.join(Gitlab['gitlab_rails']['shared_path'], 'pages'))
81: Gitlab['gitlab_pages']['artifacts_server_url'] ||= Gitlab['external_url'].chomp('/') + '/api/v4'
82:
83: pages_uri = URI(Gitlab['pages_external_url'].to_s)
84>> Gitlab['gitlab_pages']['auth_redirect_uri'] ||= pages_uri.scheme + '://projects.' + pages_uri.host + '/auth'
85: Gitlab['gitlab_pages']['auth_server'] ||= Gitlab['external_url']
86: end
87:
88: def authorize_with_gitlab
89: redirect_uri = Gitlab['gitlab_pages']['auth_redirect_uri']
90: app_name = 'GitLab Pages'
91:
92: o = query_gitlab_rails(redirect_uri, app_name)
93: if o.exitstatus.zero?
System Info:
------------
chef_version=13.6.4
platform=debian
platform_version=8.6
ruby=ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/chef-client
executable=/opt/gitlab/embedded/bin/chef-client
Running handlers:
There was an error running gitlab-ctl reconfigure:
undefined method `+' for nil:NilClass
Running handlers complete
Chef Client failed. 0 resources updated in 06 seconds
Checking PostgreSQL executables: NOT OK
Could not update PostgreSQL executables.
Shutting down all GitLab services except those needed for migrations
ok: down: alertmanager: 0s, normally up
ok: down: gitlab-monitor: 1s, normally up
ok: down: gitlab-pages: 0s, normally up
ok: down: gitlab-workhorse: 1s, normally up
ok: down: logrotate: 0s, normally up
ok: down: mailroom: 1s, normally up
ok: down: mattermost: 0s, normally up
ok: down: nginx: 1s, normally up
ok: down: node-exporter: 0s, normally up
ok: down: postgres-exporter: 1s, normally up
ok: down: prometheus: 0s, normally up
ok: down: redis-exporter: 1s, normally up
ok: down: registry: 0s, normally up
ok: down: sidekiq: 0s, normally up
ok: down: unicorn: 0s, normally up
Ensuring the required services are running
ok: run: postgresql: (pid 865) 10547s
ok: run: redis: (pid 864) 10547s
ok: run: gitaly: (pid 868) 10547s
run: postgresql: (pid 865) 10547s; run: log: (pid 782) 10550s
run: redis: (pid 864) 10547s; run: log: (pid 783) 10550s
run: gitaly: (pid 868) 10547s; run: log: (pid 788) 10550s
Reconfiguring GitLab to apply migrations
Starting Chef Client, version 13.6.4
resolving cookbooks for run list: ["gitlab-ee"]
Synchronizing Cookbooks:
- package (0.1.0)
- gitlab-ee (0.0.1)
- consul (0.1.0)
- gitlab (0.0.1)
- runit (4.3.0)
- postgresql (0.1.0)
- redis (0.1.0)
- registry (0.1.0)
- mattermost (0.1.0)
- nginx (0.1.0)
- letsencrypt (0.1.0)
- gitaly (0.1.0)
- crond (0.1.0)
- acme (3.1.0)
- compat_resource (12.19.1)
- repmgr (0.1.0)
Installing Cookbook Gems:
Compiling Cookbooks...
================================================================================
Recipe Compile Error in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab-ee/recipes/default.rb
================================================================================
NoMethodError
-------------
undefined method `+' for nil:NilClass
Cookbook Trace:
---------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/gitlab_pages.rb:84:in `parse_gitlab_pages_daemon'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/gitlab_pages.rb:28:in `parse_variables'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/settings_helper.rb:174:in `block in generate_config'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/settings_helper.rb:172:in `each'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/settings_helper.rb:172:in `generate_config'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/config.rb:26:in `from_file'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/default.rb:26:in `from_file'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab-ee/recipes/default.rb:20:in `from_file'
Relevant File Content:
----------------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/gitlab_pages.rb:
77: Gitlab['gitlab_pages']['cert_key'] ||= "/etc/gitlab/ssl/#{Gitlab['gitlab_pages']['domain']}.key"
78: end
79:
80: Gitlab['gitlab_pages']['pages_root'] ||= (Gitlab['gitlab_rails']['pages_path'] || File.join(Gitlab['gitlab_rails']['shared_path'], 'pages'))
81: Gitlab['gitlab_pages']['artifacts_server_url'] ||= Gitlab['external_url'].chomp('/') + '/api/v4'
82:
83: pages_uri = URI(Gitlab['pages_external_url'].to_s)
84>> Gitlab['gitlab_pages']['auth_redirect_uri'] ||= pages_uri.scheme + '://projects.' + pages_uri.host + '/auth'
85: Gitlab['gitlab_pages']['auth_server'] ||= Gitlab['external_url']
86: end
87:
88: def authorize_with_gitlab
89: redirect_uri = Gitlab['gitlab_pages']['auth_redirect_uri']
90: app_name = 'GitLab Pages'
91:
92: o = query_gitlab_rails(redirect_uri, app_name)
93: if o.exitstatus.zero?
System Info:
------------
chef_version=13.6.4
platform=debian
platform_version=8.6
ruby=ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/chef-client
executable=/opt/gitlab/embedded/bin/chef-client
Running handlers:
There was an error running gitlab-ctl reconfigure:
undefined method `+' for nil:NilClass
Running handlers complete
Chef Client failed. 0 resources updated in 08 seconds
Checking for an omnibus managed postgresql: OK
Checking for a newer version of PostgreSQL to install
No new version of PostgreSQL installed, nothing to upgrade to
Ensuring PostgreSQL is updated: OK
Restarting previously running GitLab services
ok: run: alertmanager: (pid 54759) 1s
ok: run: gitaly: (pid 868) 10558s
ok: run: gitlab-monitor: (pid 54771) 0s
ok: run: gitlab-pages: (pid 54780) 0s
ok: run: gitlab-workhorse: (pid 54782) 0s
ok: run: logrotate: (pid 54785) 0s
ok: run: mailroom: (pid 54813) 1s
ok: run: mattermost: (pid 54816) 0s
ok: run: nginx: (pid 54830) 1s
ok: run: node-exporter: (pid 54838) 0s
ok: run: postgres-exporter: (pid 54840) 1s
ok: run: postgresql: (pid 865) 10561s
ok: run: prometheus: (pid 54852) 0s
ok: run: redis: (pid 864) 10561s
ok: run: redis-exporter: (pid 54864) 0s
ok: run: registry: (pid 54866) 0s
ok: run: sidekiq: (pid 54873) 0s
ok: run: unicorn: (pid 54888) 0s
tput: unknown terminal "screen.xterm-256color"
_______ __ __ __
/ ____(_) /_/ / ____ _/ /_
/ / __/ / __/ / / __ `/ __ \
/ /_/ / / /_/ /___/ /_/ / /_/ /
\____/_/\__/_____/\__,_/_.___/
Upgrade complete! If your GitLab server is misbehaving try running
sudo gitlab-ctl restart
before anything else.
If you need to roll back to the previous version you can use the database
backup made during the upgrade (scroll up for the filename).(Paste any relevant logs - please use code blocks (```) to format console output, logs, and code as it's very hard to read otherwise.)
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
Unable; the command does not run:
root@zakharovite:/mnt/gitlab/backups.new# sudo gitlab-rake gitlab:env:info >/tmp/gitlab-env-info chpst: fatal: unknown user/group: git:
However, the git user exists:
root@zakharovite:/mnt/gitlab/backups.new# grep git: /etc/passwd git:x:998:998::/var/opt/gitlab:/bin/sh root@zakharovite:/mnt/gitlab/backups.new# su - git $
Results of GitLab application Check
Expand for output related to the GitLab application check
Unable; the command does not run:
root@zakharovite:/mnt/gitlab/backups.new# sudo gitlab-rake gitlab:check SANITIZE=true chpst: fatal: unknown user/group: git:
However, the git user exists:
root@zakharovite:/mnt/gitlab/backups.new# grep git: /etc/passwd git:x:998:998::/var/opt/gitlab:/bin/sh root@zakharovite:/mnt/gitlab/backups.new# su - git $
Possible fixes
None known to me.
Edited by Wayne Conrad