Error while upgrading from 15.2.2-ee to 15.3.3-ee
Summary
During the upgrade from 15.2.2-ee to 15.3.3-ee (docker), there was an error:
gitlab | Running handlers:
gitlab | [2022-09-14T20:16:20+00:00] ERROR: Running exception handlers
gitlab | There was an error running gitlab-ctl reconfigure:
gitlab |
gitlab | runit_service[gitlab-exporter] (monitoring::gitlab-exporter line 75) had an error: Mixlib::ShellOut::ShellCommandFailed: ruby_block[reload_log_service] (monitoring::gitlab-exporter line 74) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
gitlab | ---- Begin output of /opt/gitlab/embedded/bin/sv force-reload /opt/gitlab/service/gitlab-exporter/log ----
gitlab | STDOUT: kill: run: /opt/gitlab/service/gitlab-exporter/log: (pid 806) 31s, got TERM
gitlab | STDERR:
gitlab | ---- End output of /opt/gitlab/embedded/bin/sv force-reload /opt/gitlab/service/gitlab-exporter/log ----
gitlab | Ran /opt/gitlab/embedded/bin/sv force-reload /opt/gitlab/service/gitlab-exporter/log returned 1
gitlab |
gitlab | Running handlers complete
gitlab | [2022-09-14T20:16:20+00:00] ERROR: Exception handlers complete
gitlab | Infra Phase failed. 279 resources updated in 17 minutes 48 seconds
gitlab | [2022-09-14T20:16:20+00:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/chef-stacktrace.out
gitlab | [2022-09-14T20:16:20+00:00] FATAL: ---------------------------------------------------------------------------------------
gitlab | [2022-09-14T20:16:20+00:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT
gitlab | [2022-09-14T20:16:20+00:00] FATAL: ---------------------------------------------------------------------------------------
gitlab | [2022-09-14T20:16:20+00:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: runit_service[gitlab-exporter] (monitoring::gitlab-exporter line 75) had an error: Mixlib::ShellOut::ShellCommandFailed: ruby_block[reload_log_service] (monitoring::gitlab-exporter line 74) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
gitlab | ---- Begin output of /opt/gitlab/embedded/bin/sv force-reload /opt/gitlab/service/gitlab-exporter/log ----
gitlab | STDOUT: kill: run: /opt/gitlab/service/gitlab-exporter/log: (pid 806) 31s, got TERM
gitlab | STDERR:
gitlab | ---- End output of /opt/gitlab/embedded/bin/sv force-reload /opt/gitlab/service/gitlab-exporter/log ----
gitlab | Ran /opt/gitlab/embedded/bin/sv force-reload /opt/gitlab/service/gitlab-exporter/log returned 1
The system seems to work, and all background migrations are reported as completed. Still, I'd like to know if something is broken.
Relevant logs
/opt/gitlab/embedded/cookbooks/cache/chef-stacktrace.out
Generated at 2022-09-14 20:16:20 +0000 Mixlib::ShellOut::ShellCommandFailed: runit_service[gitlab-exporter] (monitoring::gitlab-exporter line 75) had an error: Mixlib::ShellOut::ShellCommandFailed: ruby_block[reload_log_service] (monitoring::gitlab-exporter line 74) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1' ---- Begin output of /opt/gitlab/embedded/bin/sv force-reload /opt/gitlab/service/gitlab-exporter/log ---- STDOUT: kill: run: /opt/gitlab/service/gitlab-exporter/log: (pid 806) 31s, got TERM STDERR: ---- End output of /opt/gitlab/embedded/bin/sv force-reload /opt/gitlab/service/gitlab-exporter/log ---- Ran /opt/gitlab/embedded/bin/sv force-reload /opt/gitlab/service/gitlab-exporter/log returned 1 /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout.rb:300:in `invalid!' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout.rb:287:in `error!' /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/helpers.rb:136:in `tap' /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/helpers.rb:136:in `safe_sv_shellout!' /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/helpers.rb:177:in `reload_log_service' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/resource.rb:1342:in `method_missing' /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:77:in `block (3 levels) in ' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/provider/ruby_block.rb:31:in `block (2 levels) in ' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/mixin/why_run.rb:51:in `add_action' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/provider.rb:293:in `converge_by' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/provider/ruby_block.rb:30:in `block in ' (eval):2:in `block in action_run' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in `instance_eval' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in `compile_and_converge_action' (eval):2:in `action_run' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/provider.rb:245:in `run_action' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/resource.rb:601:in `block in run_action' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/resource.rb:628:in `with_umask' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/resource.rb:600:in `run_action' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/runner.rb:74:in `run_action' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/runner.rb:168:in `run_delayed_notification' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/runner.rb:155:in `block in run_delayed_notifications' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/runner.rb:154:in `each' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/runner.rb:154:in `run_delayed_notifications' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/runner.rb:144:in `converge' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/provider.rb:305:in `compile_and_converge_action' (eval):2:in `action_create' /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:296:in `block in ' (eval):2:in `block in action_enable' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in `instance_eval' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in `compile_and_converge_action' (eval):2:in `action_enable' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/provider.rb:245:in `run_action' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/resource.rb:601:in `block in run_action' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/resource.rb:628:in `with_umask' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/resource.rb:600:in `run_action' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/runner.rb:74:in `run_action' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `block in run_all_actions' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `each' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `run_all_actions' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/runner.rb:132:in `block in converge' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/resource_collection/resource_list.rb:96:in `block in execute_each_resource' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:114:in `call_iterator_block' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:85:in `step' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:103:in `iterate' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:54:in `each_with_index' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/resource_collection/resource_list.rb:94:in `execute_each_resource' /opt/gitlab/embedded/lib/ruby/2.7.0/forwardable.rb:235:in `execute_each_resource' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/runner.rb:130:in `converge' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/client.rb:686:in `block in converge' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/client.rb:681:in `catch' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/client.rb:681:in `converge' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/client.rb:705:in `converge_and_save' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/client.rb:285:in `run' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/application.rb:305:in `run_with_graceful_exit_option' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/application.rb:281:in `block in run_chef_client' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/local_mode.rb:42:in `with_server_connectivity' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/application.rb:264:in `run_chef_client' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/application/base.rb:352:in `run_application' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/application.rb:67:in `run' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-bin-17.10.0/bin/cinc-client:25:in `' /opt/gitlab/embedded/bin/cinc-client:23:in `load' /opt/gitlab/embedded/bin/cinc-client:23:in `'Caused by Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1' ---- Begin output of /opt/gitlab/embedded/bin/sv force-reload /opt/gitlab/service/gitlab-exporter/log ---- STDOUT: kill: run: /opt/gitlab/service/gitlab-exporter/log: (pid 806) 31s, got TERM STDERR: ---- End output of /opt/gitlab/embedded/bin/sv force-reload /opt/gitlab/service/gitlab-exporter/log ---- Ran /opt/gitlab/embedded/bin/sv force-reload /opt/gitlab/service/gitlab-exporter/log returned 1 /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout.rb:300:in
invalid!' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout.rb:287:in
error!' /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/helpers.rb:136:intap' /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/helpers.rb:136:in
safe_sv_shellout!' /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/helpers.rb:177:inreload_log_service' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/resource.rb:1342:in
method_missing' /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:77:inblock (3 levels) in <class:RunitService>' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/provider/ruby_block.rb:31:in
block (2 levels) in class:RubyBlock' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/mixin/why_run.rb:51:inadd_action' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/provider.rb:293:in
converge_by' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/provider/ruby_block.rb:30:inblock in <class:RubyBlock>' (eval):2:in
block in action_run' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/provider.rb:304:ininstance_eval' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in
compile_and_converge_action' (eval):2:inaction_run' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/provider.rb:245:in
run_action' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/resource.rb:601:inblock in run_action' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/resource.rb:628:in
with_umask' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/resource.rb:600:inrun_action' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/runner.rb:74:in
run_action' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/runner.rb:168:inrun_delayed_notification' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/runner.rb:155:in
block in run_delayed_notifications' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/runner.rb:154:ineach' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/runner.rb:154:in
run_delayed_notifications' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/runner.rb:144:inconverge' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/provider.rb:305:in
compile_and_converge_action' (eval):2:inaction_create' /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:296:in
block in class:RunitService' (eval):2:inblock in action_enable' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in
instance_eval' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/provider.rb:304:incompile_and_converge_action' (eval):2:in
action_enable' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/provider.rb:245:inrun_action' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/resource.rb:601:in
block in run_action' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/resource.rb:628:inwith_umask' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/resource.rb:600:in
run_action' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/runner.rb:74:inrun_action' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in
block in run_all_actions' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/runner.rb:108:ineach' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in
run_all_actions' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/runner.rb:132:inblock in converge' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/resource_collection/resource_list.rb:96:in
block in execute_each_resource' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:114:incall_iterator_block' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:85:in
step' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:103:initerate' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:54:in
each_with_index' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/resource_collection/resource_list.rb:94:inexecute_each_resource' /opt/gitlab/embedded/lib/ruby/2.7.0/forwardable.rb:235:in
execute_each_resource' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/runner.rb:130:inconverge' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/client.rb:686:in
block in converge' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/client.rb:681:incatch' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/client.rb:681:in
converge' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/client.rb:705:inconverge_and_save' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/client.rb:285:in
run' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/application.rb:305:inrun_with_graceful_exit_option' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/application.rb:281:in
block in run_chef_client' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/local_mode.rb:42:inwith_server_connectivity' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/application.rb:264:in
run_chef_client' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/application/base.rb:352:inrun_application' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/application.rb:67:in
run' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-bin-17.10.0/bin/cinc-client:25:in<top (required)>' /opt/gitlab/embedded/bin/cinc-client:23:in
load' /opt/gitlab/embedded/bin/cinc-client:23:in `'
Environment details
- Operating System:
docker deployment on Rocky Linux 8.6
- Installation Target, remove incorrect values:
- Other:
docker in VM
- Other:
- Installation Type, remove incorrect values:
- Upgrade from version
15.2.2
- Upgrade from version
- Is there any other software running on the machine:
no
- Is this a single or multiple node installation?
single
- Resources
- CPU:
4 cores i7
- Memory total:
8 GB
- CPU: