**Note: I have not been able to repro this one a second time** Documenting here in case other people run into it and start searching. The error was thrown during the upgrade, during the first reconfigure run, prior to pg-upgrade. ``` Recipe: <Dynamically Defined Resource> * service[gitaly] action restart ================================================================================ Error executing action `restart` on resource 'service[gitaly]' ================================================================================ Mixlib::ShellOut::ShellCommandFailed ------------------------------------ Expected process to exit with [0], but received '1' ---- Begin output of /opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitaly ---- STDOUT: timeout: run: /opt/gitlab/service/gitaly: (pid 6484) 364s, got TERM STDERR: ---- End output of /opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitaly ---- Ran /opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitaly returned 1 Compiled Resource: ------------------ # Declared in service("gitaly") do provider Chef::Provider::Service::Simple action [:nothing] default_guard_interpreter :default service_name "gitaly" enabled nil running nil masked nil pattern "gitaly" start_command "/opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv start /opt/gitlab/service/gitaly" stop_command "/opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv stop /opt/gitlab/service/gitaly" status_command "/opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv status /opt/gitlab/service/gitaly" restart_command "/opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitaly" supports {:restart=>true, :reload=>true, :status=>true} end System Info: ------------ chef_version=13.6.4 platform=centos platform_version=6.9 ruby=ruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-linux] program_name=/opt/gitlab/embedded/bin/chef-client executable=/opt/gitlab/embedded/bin/chef-client Recipe: gitaly::enable * runit_service[gitaly] action hup - send hup to runit_service[gitaly] Recipe: <Dynamically Defined Resource> * service[gitlab-workhorse] action restart - restart service service[gitlab-workhorse] * service[gitlab-monitor] action restart - restart service service[gitlab-monitor] * service[grafana] action restart - restart service service[grafana] Running handlers: There was an error running gitlab-ctl reconfigure: service[gitaly] (dynamically defined) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1' ---- Begin output of /opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitaly ---- STDOUT: timeout: run: /opt/gitlab/service/gitaly: (pid 6484) 364s, got TERM STDERR: ---- End output of /opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitaly ---- Ran /opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitaly returned 1 Running handlers complete Chef Client failed. 73 resources updated in 03 minutes 01 seconds Ensuring PostgreSQL is updated:/opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:52:in `parse_json_file': Attributes not found in /opt/gitlab/embedded/nodes/localhost.json, has reconfigure been run yet? (GitlabCtl::Errors::NodeError) from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:64:in `get_node_attributes' from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:105:in `roles' from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:64:in `block in load_file' from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:204:in `block in add_command_under_category' from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:746:in `run' from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:31:in `<top (required)>' from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `load' from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `<main>' Ensuring PostgreSQL is updated: NOT OK Error ensuring PostgreSQL is updated. Please check the logs warning: %posttrans(gitlab-ce-12.0.0+rfbranch.114941.a52dba63-0.el6.x86_64) scriptlet failed, exit status 1 Non-fatal POSTTRANS scriptlet failure in rpm package gitlab-ce-12.0.0+rfbranch.114941.a52dba63-0.el6.x86_64 Verifying : gitlab-ce-12.0.0+rfbranch.114941.a52dba63-0.el6.x86_64 1/2 Verifying : gitlab-ce-11.11.3-ce.0.el6.x86_64 ``` At this point my instance was down. To recover I ran stop to stop gitaly, but also stop writes from being sent to it: ``` sudo gitlab-ctl stop ``` which timed out for gitaly, I then grepped and killed any gitaly processses ``` sudo ps aux | grep gitaly sudo kill -15 <pid> ``` Then ran stop again ``` sudo gitlab-ctl stop gitaly ``` Then had to clean up some remainders: ``` sudo ps aux | grep gitaly sudo kill -15 <pid> ``` Then successfully start everything: ``` sudo gitlab-ctl start gitaly ``` Then fix the other services by reconfiguring, upgprading, and restarting: ``` sudo gitlab-ctl reconfigure sudo gitlab-ctl upgrade sudo gitlab-ctl restart ``` Then I was fully upgraded and operational.