Skip to content

Database migration from 14.2.4 to 14.3.0 fails

Summary

Upgrading the docker version of GitLab from 14.2.4 to 14.3.9, the instance becomes unusable as the database migration fails.

Steps to reproduce

With a docker-compose.yml targeting gitlab/gitlab-ce:latest:

  gitlab:
    container_name: gitlab
    image: 'gitlab/gitlab-ce:latest'
    restart: always
    hostname: 'gitlab.lan'
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        external_url 'https://gitlab.lan/'
    volumes:
      - '/docker/gitlab/config:/etc/gitlab:cached'
      - '/docker/gitlab/logs:/var/log/gitlab:cached'
      - '/docker/gitlab/data:/var/opt/gitlab:cached'

Upgraded using:

docker-compose pull
docker-compose up -d

What is the current bug behavior?

Migration fails, the instance becomes unusable and is constantly restarted.

What is the expected correct behavior?

The migration works as smooth as in the past.

Relevant logs and/or screenshots

================================================================================
Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
================================================================================


Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '1'
---- Begin output of "bash"  "/tmp/chef-script20210925-27-h1fy00" ----
STDOUT: rake aborted!
StandardError: An error has occurred, all later migrations canceled:

PG::UndefinedObject: ERROR:  constraint "fk_rails_e20479742e" for table "ci_builds_metadata" does not exist
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1626:in `rename_constraint'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20210708011425_rename_ci_builds_metadata_foreign_key.rb:11:in `block in up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in `run_block'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in `block in run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in `run'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:422:in `with_lock_retries'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/v2.rb:117:in `with_lock_retries'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20210708011425_rename_ci_builds_metadata_foreign_key.rb:10:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:31:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Caused by:
ActiveRecord::StatementInvalid: PG::UndefinedObject: ERROR:  constraint "fk_rails_e20479742e" for table "ci_builds_metadata" does not exist
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1626:in `rename_constraint'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20210708011425_rename_ci_builds_metadata_foreign_key.rb:11:in `block in up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in `run_block'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in `block in run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in `run'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:422:in `with_lock_retries'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/v2.rb:117:in `with_lock_retries'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20210708011425_rename_ci_builds_metadata_foreign_key.rb:10:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:31:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Caused by:
PG::UndefinedObject: ERROR:  constraint "fk_rails_e20479742e" for table "ci_builds_metadata" does not exist
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1626:in `rename_constraint'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20210708011425_rename_ci_builds_metadata_foreign_key.rb:11:in `block in up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in `run_block'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in `block in run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in `run'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:422:in `with_lock_retries'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/v2.rb:117:in `with_lock_retries'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20210708011425_rename_ci_builds_metadata_foreign_key.rb:10:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:31:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
== 20210708011425 RenameCiBuildsMetadataForeignKey: migrating =================
-- transaction_open?()
   -> 0.0000s
-- quote_table_name("ci_builds_metadata")
   -> 0.0000s
-- quote_column_name("fk_rails_e20479742e")
   -> 0.0000s
-- quote_column_name("fk_e20479742e")
   -> 0.0000s
-- execute("ALTER TABLE \"ci_builds_metadata\"\nRENAME CONSTRAINT \"fk_rails_e20479742e\" TO \"fk_e20479742e\"\n")
STDERR: 
---- End output of "bash"  "/tmp/chef-script20210925-27-h1fy00" ----
Ran "bash"  "/tmp/chef-script20210925-27-h1fy00" returned 1

Resource Declaration:
---------------------
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb

 16:   bash "migrate #{new_resource.name} database" do
 17:     code <<-EOH
 18:     set -e
 19:     log_file="#{node['gitlab']['gitlab-rails']['log_directory']}/#{new_resource.logfile_prefix}-$(date +%Y-%m-%d-%H-%M-%S).log"
 20:     umask 077
 21:     /opt/gitlab/bin/gitlab-rake #{new_resource.rake_task} 2>& 1 | tee ${log_file}
 22:     STATUS=${PIPESTATUS[0]}
 23:     chown #{account_helper.gitlab_user}:#{account_helper.gitlab_group} ${log_file}
 24:     echo $STATUS > #{new_resource.helper.db_migrate_status_file}
 25:     exit $STATUS
 26:     EOH
 27: 
 28:     environment new_resource.environment if new_resource.property_is_set?(:environment)
 29:     new_resource.dependent_services.each do |svc|
 30:       notifies :restart, svc, :immediately
 31:     end
 32: 
 33:     not_if { new_resource.helper.migrated? }
 34:   end
 35: end

Compiled Resource:
------------------
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:16:in `block in class_from_file'

bash("migrate gitlab-rails database") do
  action [:run]
  default_guard_interpreter :default
  command nil
  backup 5
  interpreter "bash"
  declared_type :bash
  cookbook_name "gitlab"
  code "    set -e\n    log_file=\"/var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log\"\n    umask 077\n    /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}\n    STATUS=${PIPESTATUS[0]}\n    chown git:git ${log_file}\n    echo $STATUS > /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-ceec8accb09\n    exit $STATUS\n"
  domain nil
  user nil
  environment {"GITLAB_ROOT_PASSWORD"=>"…"}
  not_if { #code block }
end

System Info:
------------
chef_version=15.17.4
platform=ubuntu
platform_version=20.04
ruby=ruby 2.7.4p191 (2021-07-07 revision a21a3b7d23) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/chef-client
executable=/opt/gitlab/embedded/bin/chef-client

Possible fixes

I previously had issues with an earlier migration (#339182 (closed)) that could be fixed manually. Maybe this issue is related.