12.2.9-ee.0 to 12.3.0-ee.0 upgrade error: restart on resource 'service[gitaly]', Chef client failed.
Summary
GitLab fails to upgrade from 12.2.9-ee.0 to 12.3.0-ee.0.
I had success from 12.2.5 -> 12.2.6 -> 12.2.7 -> 12.2.8 -> 12.2.9, but then got stuck trying to upgrade to 12.3.0.
Now after it has crashed and restarted 30 times, the container is up "healthy", but GitLab shows 503. Attaching to the container keeps printing this every now and then:
==> /var/log/gitlab/gitaly/current <==
./run: 11: ulimit: error setting limit (Operation not permitted)
./run: 11: ulimit: error setting limit (Operation not permitted)
./run: 11: ulimit: error setting limit (Operation not permitted)
Our GitLab setup is an omnibus installation running on AWS, the upgrade is done by changing the release in the CloudFormation yaml.
I've previously tried to upgrade from 12.2.5 to perhaps 12.2.6 and then to 12.3.x, getting stuck in a state where GitLab was up, but 503'd on some pages, perhaps due to missing secrets.
Relevant logs and/or screenshots
Console log from GitLab container as it tries to start
Recipe: <Dynamically Defined Resource>
* service[gitaly] action restart
================================================================================
Error executing action `restart` on resource 'service[gitaly]'
================================================================================
Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '1'
---- Begin output of /opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitaly ----
STDOUT: timeout: down: /opt/gitlab/service/gitaly: 0s, normally up, want up
STDERR:
---- End output of /opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitaly ----
Ran /opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitaly returned 1
Compiled Resource:
------------------
# Declared in
service("gitaly") do
provider Chef::Provider::Service::Simple
action [:nothing] default_guard_interpreter :default
service_name "gitaly" enabled nil running nil masked nil
pattern "gitaly"
start_command "/opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv start /opt/gitlab/service/gitaly"
stop_command "/opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv stop /opt/gitlab/service/gitaly"
status_command "/opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv status /opt/gitlab/service/gitaly"
restart_command "/opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitaly"
supports {:restart=>true, :reload=>true, :status=>true}
end
System Info:
------------
chef_version=14.13.11
platform=ubuntu
platform_version=16.04
ruby=ruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/chef-client
executable=/opt/gitlab/embedded/bin/chef-client
* service[gitlab-workhorse] action restart
- restart service service[gitlab-workhorse]
* service[registry] action restart
- restart service service[registry]
* service[redis-exporter] action restart
- restart service service[redis-exporter]
* service[alertmanager] action restart
- restart service service[alertmanager]
* service[grafana] action restart
- restart service service[grafana]
Running handlers:
Running handlers complete
There was an error running gitlab-ctl reconfigure:
service[gitaly] (dynamically defined) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of /opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitaly ----
STDOUT: timeout: down: /opt/gitlab/service/gitaly: 0s, normally up, want up
STDERR:
---- End output of /opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitaly ----
Ran /opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitaly returned 1
Chef Client failed. 230 resources updated in 01 minutes 33 seconds
Results of GitLab environment info
Expand for output related to GitLab environment info
docker exec -it 135 gitlab-rake gitlab:env:info
System information
System:
Proxy: no
Current User: git
Using RVM: no
Ruby Version: 2.6.3p62
Gem Version: 2.7.9
Bundler Version:1.17.3
Rake Version: 12.3.2
Redis Version: 3.2.12
Git Version: 2.22.0
Sidekiq Version:5.2.7
Go Version: unknown
GitLab information
Version: 12.3.0-ee
Revision: c5ab21eda57
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 9.6.11
URL: https://###############
HTTP Clone URL: https://###############/some-group/some-project.git
SSH Clone URL: ssh://git@###############:443/some-group/some-project.git
Elasticsearch: no
Geo: no
Using LDAP: no
Using Omniauth: yes
Omniauth Providers:
GitLab Shell
Version: 10.0.0
Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
GitLab application check
docker exec -it 135 gitlab-rake gitlab:check SANITIZE=true
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 10.0.0 ? ... OK (10.0.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... FAIL: 14:Connect Failed
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet)
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
1/1 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.3)
Git version >= 2.22.0 ? ... yes (2.22.0)
Git user has default SSH configuration? ... yes
Active users: ... 1
Is authorized keys file accessible? ... yes
Elasticsearch version 5.6 - 6.x? ... skipped (elasticsearch is disabled)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished