Migration error on upgrading from 14.10 to 15.0 using Docker image

Summary

I can't upgrade from Gitlab (EE) 14.10 to 15.0 through Docker image because of database migrations. Command gitlab-ctl reconfigure fails with rails migration error (see below). Thanks for your help.

Steps to reproduce

From a Docker installation with gitlab/gitlab-ee:14.10.3-ee.0, I replace my configuration with gitlab/gitlab-ee:15.0.3-ee.0 and recreate the container.

Docker command I use:

sudo docker run -d --restart always \
    -p 6118:22 \
    -p 4567:4567 \
    --expose 443 \
    --hostname git.hidden.ch \
    --name gitlab \
    --volume /etc/letsencrypt/links:/etc/gitlab/ssl \
    --volume /etc/letsencrypt/:/etc/letsencrypt/ \
    --volume $PWD/config:/etc/gitlab \
    --volume $PWD/logs:/var/log/gitlab \
    --volume $PWD/data:/var/opt/gitlab \
    --volume $HOME/backups/gitlab/secret:/secret/gitlab/backups \
    --volume $PWD/lfs-storage:/mnt/storage/lfs-objects \
    --network web \
    gitlab/gitlab-ee:15.0.3-ee.0

What is the current bug behavior?

The container tries to reconfigure, fails and restart continuously.

What is the expected correct behavior?

The container is reconfigured successfully and starts with the new Docker image (gitlab/gitlab-ee:15.0.3-ee.0).

Relevant logs

End of reconfigure command logs 
Recipe: gitlab::database_migrations
  * ruby_block[check remote PG version] action nothing (skipped due to action :nothing)
  * rails_migration[gitlab-rails] action run
    * bash[migrate gitlab-rails database] action run



  ================================================================================
  Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
  ================================================================================
  
  Mixlib::ShellOut::ShellCommandFailed
  ------------------------------------
  Command execution failed. STDOUT/STDERR suppressed for sensitive resource
  
  Resource Declaration:
  ---------------------
  suppressed sensitive resource output
  
  Compiled Resource:
  ------------------
  suppressed sensitive resource output
  
  System Info:
  ------------
  chef_version=15.17.4
  platform=ubuntu
  platform_version=20.04
  ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
  program_name=/opt/gitlab/embedded/bin/cinc-client
  executable=/opt/gitlab/embedded/bin/cinc-client
  

================================================================================
Error executing action `run` on resource 'rails_migration[gitlab-rails]'
================================================================================

Mixlib::ShellOut::ShellCommandFailed
------------------------------------
bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource

Resource Declaration:
---------------------
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb

 51: rails_migration "gitlab-rails" do
 52:   rake_task 'gitlab:db:configure'
 53:   logfile_prefix 'gitlab-rails-db-migrate'
 54:   helper migration_helper
 55: 
 56:   environment env_variables
 57:   dependent_services dependent_services
 58:   notifies :run, "execute[clear the gitlab-rails cache]", :immediately
 59:   notifies :run, "ruby_block[check remote PG version]", :immediately
 60: 
 61:   only_if { migration_helper.attributes_node['auto_migrate'] }
 62: end

Compiled Resource:
------------------
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:51:in `from_file'

rails_migration("gitlab-rails") do
  action [:run]
  default_guard_interpreter :default
  declared_type :rails_migration
  cookbook_name "gitlab"
  recipe_name "database_migrations"
  rake_task "gitlab:db:configure"
  logfile_prefix "gitlab-rails-db-migrate"
  helper "*sensitive value suppressed*"
  environment "*sensitive value suppressed*"
  dependent_services []
  only_if { #code block }
end

System Info:
------------
chef_version=15.17.4
platform=ubuntu
platform_version=20.04
ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/cinc-client
executable=/opt/gitlab/embedded/bin/cinc-client





Recipe: gitlab::mailroom

  • runit_service[mailroom] action restart

    ================================================================================ Error executing action restart on resource 'runit_service[mailroom]'

    Mixlib::ShellOut::ShellCommandFailed

    Expected process to exit with [0], but received '1' ---- Begin output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/mailroom ---- STDOUT: fail: /opt/gitlab/service/mailroom: unable to change to service directory: file does not exist STDERR: ---- End output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/mailroom ---- Ran /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/mailroom returned 1

    Cookbook Trace:

    /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/helpers.rb:136:in tap' /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/helpers.rb:136:in safe_sv_shellout!' /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/helpers.rb:164:in restart_service' /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:358:in block in class:RunitService'

    Resource Declaration:

    In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/mailroom.rb

    34: runit_service 'mailroom' do 35: finish true 36: options({ 37: user: user, 38: groupname: group, 39: log_directory: mailroom_log_dir, 40: mail_room_config: mail_room_config, 41: exit_log_format: exit_log_format, 42: working_dir: mailroom_working_dir 43: }.merge(params)) 44: log_options node['gitlab']['logging'].to_hash.merge(node['gitlab']['mailroom'].to_hash) 45: end 46:

    Compiled Resource:

    Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/mailroom.rb:34:in `from_file'

    runit_service("mailroom") do action [:enable] default_guard_interpreter :default declared_type :runit_service cookbook_name "gitlab" recipe_name "mailroom" service_name "mailroom" options {:env_dir=>"/opt/gitlab/sv/mailroom/env", :user=>"git", :groupname=>"git", :log_directory=>"/var/log/gitlab/mailroom", :mail_room_config=>"/opt/gitlab/embedded/service/gitlab-rails/config/mail_room.yml", :exit_log_format=>"plain", :working_dir=>"/var/opt/gitlab/gitlab-rails/working"} log_options {"svlogd_size"=>209715200, "svlogd_num"=>30, "svlogd_timeout"=>86400, "svlogd_filter"=>"gzip", "svlogd_udp"=>nil, "svlogd_prefix"=>nil, "udp_log_shipping_host"=>nil, "udp_log_shipping_hostname"=>nil, "udp_log_shipping_port"=>514, "logrotate_frequency"=>"daily", "logrotate_maxsize"=>nil, "logrotate_size"=>nil, "logrotate_rotate"=>30, "logrotate_compress"=>"compress", "logrotate_method"=>"copytruncate", "logrotate_postrotate"=>nil, "logrotate_dateformat"=>nil, "enable"=>true, "ha"=>false, "log_directory"=>"/var/log/gitlab/mailroom", "exit_log_format"=>"plain", "incoming_email_auth_token"=>nil, "service_desk_email_auth_token"=>nil} run_template_name "mailroom" finish true log_template_name "mailroom" check_script_template_name "mailroom" finish_script_template_name "mailroom" control_template_names {} end

    System Info:

    chef_version=15.17.4 platform=ubuntu platform_version=20.04 ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux] program_name=/opt/gitlab/embedded/bin/cinc-client executable=/opt/gitlab/embedded/bin/cinc-client

Recipe: gitlab::gitlab-rails

  • execute[clear the gitlab-rails cache] action run
    • execute /opt/gitlab/bin/gitlab-rake cache:clear

Running handlers: There was an error running gitlab-ctl reconfigure:

Multiple failures occurred:

  • Mixlib::ShellOut::ShellCommandFailed occurred in Cinc Client run: rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource
  • Mixlib::ShellOut::ShellCommandFailed occurred in delayed notification: runit_service[mailroom] (gitlab::mailroom line 34) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1' ---- Begin output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/mailroom ---- STDOUT: fail: /opt/gitlab/service/mailroom: unable to change to service directory: file does not exist STDERR: ---- End output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/mailroom ---- Ran /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/mailroom returned 1

Running handlers complete Cinc Client failed. 154 resources updated in 02 minutes 21 seconds

Logs in /opt/gitlab/embedded/cookbooks/cache/chef-stacktrace.out 
Generated at 2022-06-28 08:20:45 +0000
Chef::Exceptions::MultipleFailures: Chef::Exceptions::MultipleFailures
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/provider/execute.rb:66:in `rescue in block in action_run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/provider/execute.rb:57:in `block in action_run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/mixin/why_run.rb:51:in `add_action'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/provider.rb:234:in `converge_by'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/provider/execute.rb:56:in `action_run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/provider/script.rb:60:in `action_run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/provider.rb:189:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/resource.rb:586:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/runner.rb:74:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/runner.rb:108:in `block in run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/runner.rb:108:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/runner.rb:108:in `run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/runner.rb:132:in `block in converge'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/resource_collection/resource_list.rb:96:in `block in execute_each_resource'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/resource_collection/stepable_iterator.rb:115:in `call_iterator_block'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/resource_collection/stepable_iterator.rb:86:in `step'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/resource_collection/stepable_iterator.rb:104:in `iterate'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/resource_collection/stepable_iterator.rb:55:in `each_with_index'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/resource_collection/resource_list.rb:94:in `execute_each_resource'
/opt/gitlab/embedded/lib/ruby/2.7.0/forwardable.rb:235:in `execute_each_resource'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/runner.rb:130:in `converge'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/provider.rb:246:in `compile_and_converge_action'
(eval):2:in `action_run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/provider.rb:189:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/resource.rb:586:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/runner.rb:74:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/runner.rb:108:in `block in run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/runner.rb:108:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/runner.rb:108:in `run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/runner.rb:132:in `block in converge'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/resource_collection/resource_list.rb:96:in `block in execute_each_resource'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/resource_collection/stepable_iterator.rb:115:in `call_iterator_block'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/resource_collection/stepable_iterator.rb:86:in `step'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/resource_collection/stepable_iterator.rb:104:in `iterate'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/resource_collection/stepable_iterator.rb:55:in `each_with_index'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/resource_collection/resource_list.rb:94:in `execute_each_resource'
/opt/gitlab/embedded/lib/ruby/2.7.0/forwardable.rb:235:in `execute_each_resource'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/runner.rb:130:in `converge'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/client.rb:712:in `block in converge'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/client.rb:707:in `catch'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/client.rb:707:in `converge'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/client.rb:731:in `converge_and_save'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/client.rb:288:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/application.rb:320:in `run_with_graceful_exit_option'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/application.rb:296:in `block in run_chef_client'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/local_mode.rb:42:in `with_server_connectivity'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/application.rb:279:in `run_chef_client'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/application/base.rb:330:in `run_application'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/application.rb:68:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-bin-15.17.4/bin/cinc-client:25:in `'
/opt/gitlab/embedded/bin/cinc-client:23:in `load'
/opt/gitlab/embedded/bin/cinc-client:23:in `'









Caused by Mixlib::ShellOut::ShellCommandFailed: rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/provider/execute.rb:66:in rescue in block in action_run' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/provider/execute.rb:57:in block in action_run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/mixin/why_run.rb:51:in add_action' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/provider.rb:234:in converge_by'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/provider/execute.rb:56:in action_run' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/provider/script.rb:60:in action_run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/provider.rb:189:in run_action' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/resource.rb:586:in run_action'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/runner.rb:74:in run_action' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/runner.rb:108:in block in run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/runner.rb:108:in each' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/runner.rb:108:in run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/runner.rb:132:in block in converge' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/resource_collection/resource_list.rb:96:in block in execute_each_resource'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/resource_collection/stepable_iterator.rb:115:in call_iterator_block' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/resource_collection/stepable_iterator.rb:86:in step'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/resource_collection/stepable_iterator.rb:104:in iterate' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/resource_collection/stepable_iterator.rb:55:in each_with_index'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/resource_collection/resource_list.rb:94:in execute_each_resource' /opt/gitlab/embedded/lib/ruby/2.7.0/forwardable.rb:235:in execute_each_resource'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/runner.rb:130:in converge' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/provider.rb:246:in compile_and_converge_action'
(eval):2:in action_run' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/provider.rb:189:in run_action'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/resource.rb:586:in run_action' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/runner.rb:74:in run_action'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/runner.rb:108:in block in run_all_actions' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/runner.rb:108:in each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/runner.rb:108:in run_all_actions' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/runner.rb:132:in block in converge'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/resource_collection/resource_list.rb:96:in block in execute_each_resource' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/resource_collection/stepable_iterator.rb:115:in call_iterator_block'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/resource_collection/stepable_iterator.rb:86:in step' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/resource_collection/stepable_iterator.rb:104:in iterate'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/resource_collection/stepable_iterator.rb:55:in each_with_index' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/resource_collection/resource_list.rb:94:in execute_each_resource'
/opt/gitlab/embedded/lib/ruby/2.7.0/forwardable.rb:235:in execute_each_resource' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/runner.rb:130:in converge'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/client.rb:712:in block in converge' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/client.rb:707:in catch'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/client.rb:707:in converge' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/client.rb:731:in converge_and_save'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/client.rb:288:in run' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/application.rb:320:in run_with_graceful_exit_option'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/application.rb:296:in block in run_chef_client' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/local_mode.rb:42:in with_server_connectivity'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/application.rb:279:in run_chef_client' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/application/base.rb:330:in run_application'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.17.4/lib/chef/application.rb:68:in run' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-bin-15.17.4/bin/cinc-client:25:in <top (required)>'
/opt/gitlab/embedded/bin/cinc-client:23:in load' /opt/gitlab/embedded/bin/cinc-client:23:in '
gitlab-rails-db-migrate.log 
rake aborted!
StandardError: An error has occurred, all later migrations canceled:

Gitlab::Database::BackgroundMigration::BatchedMigrationRunner::FailedToFinalize
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:76:in finalize' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:10:in finalize'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:139:in finalize_batched_background_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:962:in ensure_batched_background_migration_is_finished'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20220213103859_remove_integrations_type.rb:48:in up' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:115:in configure_database' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:95:in block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in '


Caused by:
Gitlab::Database::BackgroundMigration::BatchedMigrationRunner::FailedToFinalize: Gitlab::Database::BackgroundMigration::BatchedMigrationRunner::FailedToFinalize
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:76:in finalize' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:10:in finalize'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:139:in finalize_batched_background_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:962:in ensure_batched_background_migration_is_finished'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20220213103859_remove_integrations_type.rb:48:in up' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:115:in configure_database' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:95:in block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in '
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
== 20220213103859 RemoveIntegrationsType: migrating ===========================

Details of package version

Omnibus with Docker

Environment details

  • Operating System: Docker image found on Docker Hub (gitlab/gitlab-ee:15.0.3-ee.0)
  • Installation Target, remove incorrect values:
    • Docker container on self hosted server
  • Installation Type, remove incorrect values:
    • Upgrade from version 14.10.3
  • Is there any other software running on the machine: no
  • Is this a single or multiple node installation?: Single node
  • Resources
    • CPU: x86_64 AMD Sempron(tm) 145 Processor
    • Memory total: 16Go

Configuration details

"/etc/gitlab/gitlab.rb" 
external_url 'https://HIDDEN'
gitlab_rails['gitlab_email_enabled'] = true
gitlab_rails['gitlab_email_from'] = 'HIDDEN'
gitlab_rails['gitlab_email_display_name'] = 'Gitlab'
gitlab_rails['gitlab_email_reply_to'] = 'webmasters@hidden.ch'
gitlab_rails['incoming_email_enabled'] = true
gitlab_rails['incoming_email_address'] = "gitlab+%{key}@hidden.ch"
gitlab_rails['incoming_email_email'] = "tim@hidden.ch"
gitlab_rails['incoming_email_password'] = "HIDDEN"
gitlab_rails['incoming_email_host'] = "imap.gmail.com"
gitlab_rails['incoming_email_port'] = 993
gitlab_rails['incoming_email_ssl'] = true
gitlab_rails['incoming_email_start_tls'] = false
gitlab_rails['incoming_email_mailbox_name'] = "Gitlab"
gitlab_rails['incoming_email_log_directory'] = "/var/log/gitlab/mailroom"
gitlab_rails['lfs_enabled'] = true
gitlab_rails['lfs_storage_path'] = "/mnt/storage/lfs-objects"
gitlab_rails['omniauth_enabled'] = true
gitlab_rails['omniauth_external_providers'] = ['google_oauth2']
gitlab_rails['omniauth_providers'] = [HIDDEN]
gitlab_rails['backup_archive_permissions'] = 0644
gitlab_rails['backup_upload_connection'] = {HIDDEN}
gitlab_rails['backup_upload_remote_directory'] = 'backups-gitlab'
gitlab_rails['gitlab_shell_ssh_port'] = 6118
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp.gmail.com"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_user_name'] = "HIDDEN"
gitlab_rails['smtp_password'] = "HIDDEN"
gitlab_rails['smtp_domain'] = "smtp.gmail.com"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_tls'] = false
gitlab_rails['smtp_openssl_verify_mode'] = 'peer'
registry_external_url 'https://git.hidden.ch:4567'
gitlab_rails['registry_enabled'] = true
nginx['client_max_body_size'] = '500m'
nginx['client_body_buffer_size'] = '1M';
Edited by Tim Izzo