Failure to update from 16.2.4 to 16.3
Summary
Updating the GitLab Omnibus docker image from 16.2.4 to 16.3 results in a failure to start.
Steps to reproduce
- Run GitLab 16.2.4 using the official
gitlab/gitlab-ce:16.2.4-ce.0docker image. - Run
docker compose down - Change the docker image to
gitlab/gitlab-ce:16.3.0-ce.0(currently same as:latest) - Run
docker compose up -d - Run
docker compose logs -f - Fear, surprise, 404.
What is the current bug behavior?
The gitlab-ctl reconfigure script that runs as the docker container starts up fails with this error message:
================================================================================
Error executing action `run` on resource 'bash_hide_env[migrate gitlab-rails database]'
================================================================================
What is the expected correct behavior?
I expect the usual amounts of happiness and joy as I read the outputs of docker compose logs -f after an upgrade.
Relevant logs
cinc-stacktrace.out
Generated at 2023-08-23 09:09:23 +0000 Mixlib::ShellOut::ShellCommandFailed: rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 20) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1' ---- Begin output of "bash" ---- STDOUT: rake aborted! ArgumentError: is not a recognized provider /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/content_security_policy/config_loader.rb:197:in `build_lfs_url' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/content_security_policy/config_loader.rb:89:in `allow_lfs' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/content_security_policy/config_loader.rb:24:in `default_directives' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/content_security_policy/config_loader.rb:206:in `initialize' /opt/gitlab/embedded/service/gitlab-rails/config/initializers/content_security_policy.rb:12:in `new' /opt/gitlab/embedded/service/gitlab-rails/config/initializers/content_security_policy.rb:12:in `block in <top (required)>' /opt/gitlab/embedded/service/gitlab-rails/config/initializers/content_security_policy.rb:11:in `<top (required)>' /opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>' <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:38:in `require' <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:38:in `require' /opt/gitlab/embedded/bin/bundle:25:in `load' /opt/gitlab/embedded/bin/bundle:25:in `<main>' Tasks: TOP => gitlab:db:configure => environment (See full trace by running task with --trace) STDERR: ---- End output of "bash" ---- Ran "bash" returned 1 /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout.rb:300:in `invalid!' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout.rb:287:in `error!' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout/helper.rb:130:in `shell_out_compacted!' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout/helper.rb:54:in `shell_out!' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider/execute.rb:52:in `block (2 levels) in <class:Execute>' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/mixin/why_run.rb:51:in `add_action' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:293:in `converge_by' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider/execute.rb:50:in `block in <class:Execute>' (eval):2:in `block in action_run' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in `instance_eval' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in `compile_and_converge_action' (eval):2:in `action_run' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:245:in `run_action' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:601:in `block in run_action' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:628:in `with_umask' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:600:in `run_action' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:74:in `run_action' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `block in run_all_actions' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `each' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `run_all_actions' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection.rb:64:in `insert' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/dsl/declare_resource.rb:267:in `declare_resource' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/dsl/resources.rb:36:in `bash_hide_env' /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:20:in `block in class_from_file' (eval):2:in `block in action_run' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in `instance_eval' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in `compile_and_converge_action' (eval):2:in `action_run' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:245:in `run_action' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:601:in `block in run_action' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:628:in `with_umask' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:600:in `run_action' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:74:in `run_action' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `block in run_all_actions' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `each' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `run_all_actions' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:132:in `block in converge' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/resource_list.rb:96:in `block in execute_each_resource' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:114:in `call_iterator_block' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:85:in `step' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:103:in `iterate' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:54:in `each_with_index' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/resource_list.rb:94:in `execute_each_resource' /opt/gitlab/embedded/lib/ruby/3.0.0/forwardable.rb:238:in `execute_each_resource' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:130:in `converge' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:692:in `block in converge' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:687:in `catch' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:687:in `converge' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:711:in `converge_and_save' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:285:in `run' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:305:in `run_with_graceful_exit_option' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:281:in `block in run_chef_client' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/local_mode.rb:42:in `with_server_connectivity' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:264:in `run_chef_client' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application/base.rb:352:in `run_application' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:67:in `run' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-bin-17.10.0/bin/cinc-client:25:in `<top (required)>' /opt/gitlab/embedded/bin/cinc-client:25:in `load' /opt/gitlab/embedded/bin/cinc-client:25:in `<main>' >>>> Caused by Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1' ---- Begin output of "bash" ---- STDOUT: rake aborted! ArgumentError: is not a recognized provider /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/content_security_policy/config_loader.rb:197:in `build_lfs_url' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/content_security_policy/config_loader.rb:89:in `allow_lfs' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/content_security_policy/config_loader.rb:24:in `default_directives' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/content_security_policy/config_loader.rb:206:in `initialize' /opt/gitlab/embedded/service/gitlab-rails/config/initializers/content_security_policy.rb:12:in `new' /opt/gitlab/embedded/service/gitlab-rails/config/initializers/content_security_policy.rb:12:in `block in <top (required)>' /opt/gitlab/embedded/service/gitlab-rails/config/initializers/content_security_policy.rb:11:in `<top (required)>' /opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>' <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:38:in `require' <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:38:in `require' /opt/gitlab/embedded/bin/bundle:25:in `load' /opt/gitlab/embedded/bin/bundle:25:in `<main>' Tasks: TOP => gitlab:db:configure => environment (See full trace by running task with --trace) STDERR: ---- End output of "bash" ---- Ran "bash" returned 1 /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout.rb:300:in `invalid!' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout.rb:287:in `error!' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout/helper.rb:130:in `shell_out_compacted!' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout/helper.rb:54:in `shell_out!' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider/execute.rb:52:in `block (2 levels) in <class:Execute>' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/mixin/why_run.rb:51:in `add_action' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:293:in `converge_by' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider/execute.rb:50:in `block in <class:Execute>' (eval):2:in `block in action_run' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in `instance_eval' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in `compile_and_converge_action' (eval):2:in `action_run' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:245:in `run_action' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:601:in `block in run_action' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:628:in `with_umask' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:600:in `run_action' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:74:in `run_action' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `block in run_all_actions' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `each' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `run_all_actions' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection.rb:64:in `insert' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/dsl/declare_resource.rb:267:in `declare_resource' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/dsl/resources.rb:36:in `bash_hide_env' /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:20:in `block in class_from_file' (eval):2:in `block in action_run' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in `instance_eval' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in `compile_and_converge_action' (eval):2:in `action_run' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:245:in `run_action' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:601:in `block in run_action' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:628:in `with_umask' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:600:in `run_action' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:74:in `run_action' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `block in run_all_actions' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `each' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `run_all_actions' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:132:in `block in converge' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/resource_list.rb:96:in `block in execute_each_resource' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:114:in `call_iterator_block' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:85:in `step' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:103:in `iterate' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:54:in `each_with_index' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/resource_list.rb:94:in `execute_each_resource' /opt/gitlab/embedded/lib/ruby/3.0.0/forwardable.rb:238:in `execute_each_resource' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:130:in `converge' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:692:in `block in converge' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:687:in `catch' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:687:in `converge' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:711:in `converge_and_save' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:285:in `run' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:305:in `run_with_graceful_exit_option' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:281:in `block in run_chef_client' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/local_mode.rb:42:in `with_server_connectivity' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:264:in `run_chef_client' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application/base.rb:352:in `run_application' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:67:in `run' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-bin-17.10.0/bin/cinc-client:25:in `<top (required)>' /opt/gitlab/embedded/bin/cinc-client:25:in `load' /opt/gitlab/embedded/bin/cinc-client:25:in `<main>'
Details of package version
Provide the package version installation details
Environment details
- Operating System: Alpine 3.18
- Installation Target, remove incorrect values:
- VM: Xen
- Installation Type, remove incorrect values:
- Upgrade from version 16.2.4
- Is there any other software running on the machine: traefik
- Is this a single or multiple node installation: single
- Resources
- CPU: AMD EPYC, 8 cores
- Memory total: 12GB
Configuration details
Provide the relevant sections of `/etc/gitlab/gitlab.rb`
external_url 'https://gitlab.XXXXXXX.XXX' gitlab_rails['time_zone'] = 'Europe/Amsterdam' gitlab_rails['smtp_enable'] = true gitlab_rails['smtp_address'] = "mail.XXXXXXX.XXX" gitlab_rails['smtp_port'] = 587 gitlab_rails['smtp_user_name'] = "gitlab-admin@XXXXXXX.XXX" gitlab_rails['smtp_password'] = "XXXXXXXXXXXXXXXXXXXXXXXX" gitlab_rails['smtp_domain'] = "XXXXXXX.XXX" gitlab_rails['gitlab_email_from'] = 'gitlab-admin@XXXXXXX.XXX' gitlab_rails['gitlab_email_display_name'] = 'GitLab' gitlab_rails['gitlab_default_projects_features_issues'] = false gitlab_rails['gitlab_default_projects_features_merge_requests'] = false gitlab_rails['gitlab_default_projects_features_wiki'] = false gitlab_rails['gitlab_default_projects_features_snippets'] = false gitlab_rails['gitlab_default_projects_features_builds'] = false gitlab_rails['gitlab_default_projects_features_container_registry'] = false gitlab_rails['gravatar_plain_url'] = 'http://cdn.libravatar.org/avatar/%{hash}?s=%{size}&d=identicon' gitlab_rails['gravatar_ssl_url'] = 'https://seccdn.libravatar.org/avatar/%{hash}?s=%{size}&d=identicon' gitlab_rails['trusted_proxies'] = [ 'XXX.XXX.XXX.XXX' ] gitlab_rails['content_security_policy'] = { 'enabled' => true, } registry_external_url 'https://docker.XXXXXXX.XXX' nginx['listen_port'] = 80 nginx['listen_https'] = false pages_external_url "http://pages.XXXXXXX.XXX/" gitlab_pages['enable'] = true registry_nginx['listen_https'] = false registry_nginx['listen_port'] = 80