Docker container fails to start after upgrading from 14.9.3 to 14.10.0

Summary

Docker container fails to start after upgrading from 14.9.3 to 14.10.0. It appears to be related to database migration.

Steps to reproduce

I am using gitlab/gitlab-ce:latest. After updating with docker-compose the container keeps restarting without ever coming up. The final log message is:

There was an error running gitlab-ctl reconfigure:
rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource

Relevant logs

Relevant logs 
server_1  | Recipe: gitlab::database_migrations
server_1  |   * ruby_block[check remote PG version] action nothing (skipped due to action :nothing)
server_1  |   * rails_migration[gitlab-rails] action run
server_1  |     * bash[migrate gitlab-rails database] action run
server_1  |
server_1  |       ================================================================================
server_1  |       Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
server_1  |       ================================================================================
server_1  |
server_1  |       Mixlib::ShellOut::ShellCommandFailed
server_1  |       ------------------------------------
server_1  |       Command execution failed. STDOUT/STDERR suppressed for sensitive resource
server_1  |
server_1  |       Resource Declaration:
server_1  |       ---------------------
server_1  |       suppressed sensitive resource output
server_1  |
server_1  |       Compiled Resource:
server_1  |       ------------------
server_1  |       suppressed sensitive resource output
server_1  |
server_1  |       System Info:
server_1  |       ------------
server_1  |       chef_version=15.17.4
server_1  |       platform=ubuntu
server_1  |       platform_version=20.04
server_1  |       ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
server_1  |       program_name=/opt/gitlab/embedded/bin/chef-client
server_1  |       executable=/opt/gitlab/embedded/bin/chef-client
server_1  |
server_1  |
server_1  |     ================================================================================
server_1  |     Error executing action `run` on resource 'rails_migration[gitlab-rails]'
server_1  |     ================================================================================
server_1  |
server_1  |     Mixlib::ShellOut::ShellCommandFailed
server_1  |     ------------------------------------
server_1  |     bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error:
Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource
server_1  |
server_1  |     Resource Declaration:
server_1  |     ---------------------
server_1  |     # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb
server_1  |
server_1  |      51: rails_migration "gitlab-rails" do
server_1  |      52:   rake_task 'gitlab:db:configure'
server_1  |      53:   logfile_prefix 'gitlab-rails-db-migrate'
server_1  |      54:   helper migration_helper
server_1  |      55:
server_1  |      56:   environment env_variables
server_1  |      57:   dependent_services dependent_services
server_1  |      58:   notifies :run, "execute[clear the gitlab-rails cache]", :immediately
server_1  |      59:   notifies :run, "ruby_block[check remote PG version]", :immediately
server_1  |      60:
server_1  |      61:   only_if { migration_helper.attributes_node['auto_migrate'] }
server_1  |      62: end
server_1  |
server_1  |     Compiled Resource:
server_1  |     ------------------
server_1  |     # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:51:in `from_file'
server_1  |
server_1  |     rails_migration("gitlab-rails") do
server_1  |       action [:run]
server_1  |       default_guard_interpreter :default
server_1  |       declared_type :rails_migration
server_1  |       cookbook_name "gitlab"
server_1  |       recipe_name "database_migrations"
server_1  |       rake_task "gitlab:db:configure"
server_1  |       logfile_prefix "gitlab-rails-db-migrate"
server_1  |       helper "*sensitive value suppressed*"
server_1  |       environment "*sensitive value suppressed*"
server_1  |       dependent_services []
server_1  |       only_if { #code block }
server_1  |     end
server_1  |
server_1  |     System Info:
server_1  |     ------------
server_1  |     chef_version=15.17.4
server_1  |     platform=ubuntu
server_1  |     platform_version=20.04
server_1  |     ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
server_1  |     program_name=/opt/gitlab/embedded/bin/chef-client
server_1  |     executable=/opt/gitlab/embedded/bin/chef-client
server_1  |
server_1  |
server_1  | Running handlers:
server_1  | There was an error running gitlab-ctl reconfigure:
server_1  |
server_1  | rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource

Details of package version

Provide the package version installation details 
Thank you for using GitLab Docker Image!
Current version: gitlab-ce=14.10.0-ce.0

root@dc20044b180c:/# dpkg-query -l "gitlab-*"
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-===========================================================
ii  gitlab-ce      14.10.0-ce.0 amd64        GitLab Community Edition (including NGINX, Postgres, Redis)
un  gitlab-ee                    (no description available)

Environment details

  • Operating System: Host is Ubuntu 20.04.3 LTS
  • Installation Target, remove incorrect values:
    • Docker
  • Installation Type, remove incorrect values:
    • Upgrade from version 14.9.3
  • Is there any other software running on the machine: Traefik is acting as a reverse proxy.
  • Is this a single or multiple node installation?: Single
  • Resources
    • CPU: Docker VM has 5 cores
    • Memory total: 12GB

Configuration details

Provide the relevant sections of `/etc/gitlab/gitlab.rb`

Environment key from docker-compose.yml:

    environment:
      TZ: America/New_York
      GITLAB_OMNIBUS_CONFIG: |
        # Because we set the external url to https but traefik is managing ssl,
        # we need to manually set the port and disable LE and ssl.
        external_url 'https://git..com/'
        letsencrypt['enable'] = false
        nginx['listen_port'] = 80
        nginx['listen_https'] = false
        gitlab_rails['smtp_enable'] = true
        gitlab_rails['smtp_address'] = "smtp.gmail.com"
        gitlab_rails['smtp_port'] = 587
        gitlab_rails['smtp_user_name'] = "@gmail.com"
        gitlab_rails['smtp_password'] = File.read('/run/secrets/smtp_password')
        gitlab_rails['smtp_domain'] = "smtp.gmail.com"
        gitlab_rails['smtp_authentication'] = "login"
        gitlab_rails['smtp_enable_starttls_auto'] = true
        gitlab_rails['smtp_tls'] = false
        gitlab_rails['smtp_openssl_verify_mode'] = 'peer' # Can be: 'none', 'peer', 'client_once', 'fail_if_no_peer_cert', see http://api.rubyonrails.org/classes/ActionMailer/Base.html
        gitlab_rails['backup_keep_time'] = 2592000 # Delete backups older than 30 days (in seconds)
        gitlab_rails['time_zone'] = 'America/New_York'
        prometheus['enable'] = false # Prometheus uses a lot of disk space and we dont need it.