Database migration fails from 14.10.3 to 15.0

Summary

I'm trying to migrate my docker-based instance from 14.10.3 to 15.0 fails with an error during database migration.

Steps to reproduce

I pulled the latest image and recreated the container:

docker-compose pull
docker-compose up

What is the current bug behavior?

The migration fails with an error that doesn't provide any specifics that I could use for further, own investigation.

What is the expected correct behavior?

The migration succeeds.

Relevant logs and/or screenshots

 Recipe: gitlab::database_migrations
   * ruby_block[check remote PG version] action nothing (skipped due to action :nothing)
   * rails_migration[gitlab-rails] action run
     * bash[migrate gitlab-rails database] action run
       
       ================================================================================
       Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
       ================================================================================
       
       Mixlib::ShellOut::ShellCommandFailed
       ------------------------------------
       Command execution failed. STDOUT/STDERR suppressed for sensitive resource
       
       Resource Declaration:
       ---------------------
       suppressed sensitive resource output
       
       Compiled Resource:
       ------------------
       suppressed sensitive resource output
       
       System Info:
       ------------
       chef_version=15.17.4
       platform=ubuntu
       platform_version=20.04
       ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
       program_name=/opt/gitlab/embedded/bin/cinc-client
       executable=/opt/gitlab/embedded/bin/cinc-client
       
     
     ================================================================================
     Error executing action `run` on resource 'rails_migration[gitlab-rails]'
     ================================================================================
     
     Mixlib::ShellOut::ShellCommandFailed
     ------------------------------------
     bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for s
     
     Resource Declaration:
     ---------------------
     # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb
     
      51: rails_migration "gitlab-rails" do
      52:   rake_task 'gitlab:db:configure'
      53:   logfile_prefix 'gitlab-rails-db-migrate'
      54:   helper migration_helper
      55: 
      56:   environment env_variables
      57:   dependent_services dependent_services
      58:   notifies :run, "execute[clear the gitlab-rails cache]", :immediately
      59:   notifies :run, "ruby_block[check remote PG version]", :immediately
      60: 
      61:   only_if { migration_helper.attributes_node['auto_migrate'] }
      62: end
     
     Compiled Resource:
     ------------------
     # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:51:in `from_file'
     
     rails_migration("gitlab-rails") do
       action [:run]
       default_guard_interpreter :default
       declared_type :rails_migration
       cookbook_name "gitlab"
       recipe_name "database_migrations"
       rake_task "gitlab:db:configure"
       logfile_prefix "gitlab-rails-db-migrate"
       helper "*sensitive value suppressed*"
       environment "*sensitive value suppressed*"
       dependent_services []
       only_if { #code block }
     end
     
     System Info:
     ------------
     chef_version=15.17.4
     platform=ubuntu
     platform_version=20.04
     ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
     program_name=/opt/gitlab/embedded/bin/cinc-client
     executable=/opt/gitlab/embedded/bin/cinc-client

Results of GitLab environment info

This from within the container while it attempts to migrate from 14.3 to 15.0:

Expand for output related to GitLab environment info 


System information
System:		
Current User:	git
Using RVM:	no
Ruby Version:	2.7.5p203
Gem Version:	3.1.4
Bundler Version:2.2.33
Rake Version:	13.0.6
Redis Version:	6.2.6
Sidekiq Version:6.4.0
Go Version:	unknown

GitLab information
Version:	15.0.0
Revision:	8a186dedfc1
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	12.10
URL:		https://my.server
HTTP Clone URL:	https://my.server/some-group/some-project.git
SSH Clone URL:	git@my.server:some-group/some-project.git
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: 

GitLab Shell
Version:	14.3.0
Repository storage paths:
- default: 	/var/opt/gitlab/git-data/repositories
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell

Results of GitLab application Check

I could not capture this in the 15.0 container, because it was restarting too fast. So this is the output from the 14.10.3 version of the container:

Expand for output related to the GitLab application check 
Checking GitLab subtasks ...


Checking GitLab Shell ...


GitLab Shell: ... GitLab Shell version >= 13.25.1 ? ... OK (13.25.1)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Gitaly ...


Gitaly: ... default ... OK


Checking Gitaly ... Finished


Checking Sidekiq ...


Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1


Checking Sidekiq ... Finished


Checking Incoming Email ...


Incoming Email: ... Reply by email is disabled in config/gitlab.yml


Checking Incoming Email ... Finished


Checking LDAP ...


LDAP: ... LDAP is disabled in config/gitlab.yml


Checking LDAP ... Finished


Checking GitLab App ...


Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ...
4/1 ... yes
3/2 ... yes
3/3 ... yes
6/4 ... yes
3/5 ... yes
4/6 ... yes
4/7 ... yes
9/9 ... yes
10/11 ... yes
12/12 ... yes
9/13 ... yes
Redis version >= 5.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.5)
Git user has default SSH configuration? ... yes
Active users: ... 5
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes


Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished
Edited by Felix Schwarz