DB migration failed when upgrading from Omnibus Gitlab-CE v14.8.x to v14.10.0 on Ubuntu

Summary

DB migration failed when upgrading from Omnibus Gitlab-CE v14.8.2 to v14.10.0 on Ubuntu

Steps to reproduce

Have a Gitlab-CE v14.8.2
apt-get update ; apt-get upgrade

What is the current bug behavior?

Gitlab-rails DB:migration error
API return v14.10 but migration were not applied

What is the expected correct behavior?

Upgrade should pass without error

Relevant logs

Recipe: gitlab::database_migrations
  * ruby_block[check remote PG version] action nothing (skipped due to action :nothing)
  * rails_migration[gitlab-rails] action run
    * bash[migrate gitlab-rails database] action run

  ================================================================================
  Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
  ================================================================================

  Mixlib::ShellOut::ShellCommandFailed
  ------------------------------------
  Command execution failed. STDOUT/STDERR suppressed for sensitive resource

  Resource Declaration:
  ---------------------
  suppressed sensitive resource output

  Compiled Resource:
  ------------------
  suppressed sensitive resource output

  System Info:
  ------------
  chef_version=15.17.4
  platform=ubuntu
  platform_version=20.04
  ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
  program_name=/opt/gitlab/embedded/bin/chef-client
  executable=/opt/gitlab/embedded/bin/chef-client


================================================================================
Error executing action `run` on resource 'rails_migration[gitlab-rails]'
================================================================================

Mixlib::ShellOut::ShellCommandFailed
------------------------------------
bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource

Resource Declaration:
---------------------
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb

 51: rails_migration "gitlab-rails" do
 52:   rake_task 'gitlab:db:configure'
 53:   logfile_prefix 'gitlab-rails-db-migrate'
 54:   helper migration_helper
 55:
 56:   environment env_variables
 57:   dependent_services dependent_services
 58:   notifies :run, "execute[clear the gitlab-rails cache]", :immediately
 59:   notifies :run, "ruby_block[check remote PG version]", :immediately
 60:
 61:   only_if { migration_helper.attributes_node['auto_migrate'] }
 62: end

Compiled Resource:
------------------
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:51:in `from_file'

rails_migration("gitlab-rails") do
  action [:run]
  default_guard_interpreter :default
  declared_type :rails_migration
  cookbook_name "gitlab"
  recipe_name "database_migrations"
  rake_task "gitlab:db:configure"
  logfile_prefix "gitlab-rails-db-migrate"
  helper "*sensitive value suppressed*"
  environment "*sensitive value suppressed*"
  dependent_services ["runit_service[puma]", "sidekiq_service[sidekiq]"]
  only_if { #code block }
end

System Info:
------------
chef_version=15.17.4
platform=ubuntu
platform_version=20.04
ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/chef-client
executable=/opt/gitlab/embedded/bin/chef-client


Running handlers:
There was an error running gitlab-ctl reconfigure:
rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource
Running handlers complete
Chef Infra Client failed. 2 resources updated in 31 seconds

# /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2022-04-25-17-38-04.log
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:

Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active':       {:job_class_name=>"ProjectNamespaces::BackfillProjectNamespaces", :table_name=>:projects, :column_name=>:id, :job_arguments=>[nil, "up"]}

Finalize it manualy by running

        sudo gitlab-rake gitlab:background_migrations:finalize[ProjectNamespaces::BackfillProjectNamespaces,projects,id,'[null\,"up"]']

For more information, check the documentation

        https://docs.gitlab.com/ee/user/admin_area/monitoring/background_migrations.html#database-migrations-failing-because-of-batched-background-migration-not-finished
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:960:in `ensure_batched_background_migration_is_finished'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20220322071127_finalize_project_namespaces_backfill.rb:10:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:112:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:172:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:110:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:118:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:70:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:305:in `block in transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:304:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:115:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:95:in `block (3 levels) in '
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `'

Caused by:
Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active':       {:job_class_name=>"ProjectNamespaces::BackfillProjectNamespaces", :table_name=>:projects, :column_name=>:id, :job_arguments=>[nil, "up"]}

Finalize it manualy by running

        sudo gitlab-rake gitlab:background_migrations:finalize[ProjectNamespaces::BackfillProjectNamespaces,projects,id,'[null\,"up"]']

For more information, check the documentation

        https://docs.gitlab.com/ee/user/admin_area/monitoring/background_migrations.html#database-migrations-failing-because-of-batched-background-migration-not-finished
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:960:in `ensure_batched_background_migration_is_finished'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20220322071127_finalize_project_namespaces_backfill.rb:10:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:112:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:172:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:110:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:118:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:70:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:305:in `block in transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:304:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:115:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:95:in `block (3 levels) in '
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
== 20220322071127 FinalizeProjectNamespacesBackfill: migrating ================

Details of package version

Provide the package version installation details

# dpkg-query -l "gitlab-*"
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-===========================================================
iF  gitlab-ce      14.10.0-ce.0 amd64        GitLab Community Edition (including NGINX, Postgres, Redis)
un  gitlab-ee                    (no description available)

Environment details

Operating System: Ubuntu 20.04 amd64
Installation Target, remove incorrect values:
- VM: on-premises
Installation Type, remove incorrect values:
- Upgrade from version 14.8.2
Is there any other software running on the machine: no
Is this a single or multiple node installation? single
Resources
- CPU: 4
- Memory total: 8 GB

Configuration details

Provide the relevant sections of `/etc/gitlab/gitlab.rb`

external_url 'https://xxx.domain.tld'
gitlab_rails['time_zone'] = 'Europe/Paris'
gitlab_rails['gitlab_email_enabled'] = true
gitlab_rails['gitlab_email_from'] = 'sv+gitlab-xxx@domain.tld'
gitlab_rails['gitlab_email_display_name'] = 'Gitlab xxx.domain.tld'
gitlab_rails['gitlab_email_reply_to'] = 'sv+gitlab-xxx@domain.tld'
gitlab_rails['gitlab_default_projects_features_container_registry'] = true
gitlab_rails['lfs_enabled'] = true
gitlab_rails['ldap_enabled'] = true
gitlab_rails['ldap_servers'] = YAML.load <<-'EOS'
  main: # 'main' is the GitLab 'provider ID' of this LDAP server
    label: 'LDAP org'
    host: 'ldap.domain.tld'
    port: 389
    uid: 'uid'
    encryption: 'start_tls' # "start_tls" or "simple_tls" or "plain"
    verify_certificates: false
    active_directory: true
    allow_username_or_email_login: false
    lowercase_usernames: true
    block_auto_created_users: false
    base: 'dc=org,dc=fr'
    attributes:
      username: ['uid', 'userid', 'sAMAccountName']
      email:    ['mail', 'email', 'userPrincipalName']
      name:       'cn'
      first_name: 'givenName'
      last_name:  'sn'
EOS
gitlab_rails['backup_keep_time'] = 172800
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "localhost"
gitlab_rails['smtp_port'] = 25
gitlab_rails['smtp_domain'] = 'domain.tld'
gitlab_rails['smtp_tls'] = false
gitlab_rails['smtp_openssl_verify_mode'] = 'none'
gitlab_rails['smtp_enable_starttls_auto'] = false
gitlab_rails['smtp_ssl'] = false
gitlab_rails['smtp_force_ssl'] = false
registry_external_url 'https://xxx.domain.tld:4567'
gitlab_rails['registry_enabled'] = true
registry['registry_http_addr'] = "localhost:5000"
registry['debug_addr'] = "localhost:5001"
nginx['redirect_http_to_https'] = true
nginx['ssl_certificate'] = "/etc/ssl/certs/xxx.domain.tld.pem"
nginx['ssl_certificate_key'] = "/etc/ssl/private/xxx.domain.tld.key"
gitlab_pages['enable'] = true
gitlab_pages['access_control'] = true
pages_nginx['enable'] = true
pages_nginx['ssl_certificate'] = "/etc/gitlab/ssl/xxx.domain.tld.crt"
pages_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/xxx.domain.tld.key"
registry_nginx['enable'] = true