Skip to content

Database migration from 14.1.3 to 14.2.1 fails

Summary

Upgrading the docker version of GitLab from 14.1.3 to 14.2.1, the instance becomes unusable as the database migration fails.

Steps to reproduce

With a docker-compose.yml targeting gitlab/gitlab-ce:latest:

  gitlab:
    container_name: gitlab
    image: 'gitlab/gitlab-ce:latest'
    restart: always
    hostname: 'gitlab.lan'
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        external_url 'https://gitlab.lan/'
    volumes:
      - '/docker/gitlab/config:/etc/gitlab:cached'
      - '/docker/gitlab/logs:/var/log/gitlab:cached'
      - '/docker/gitlab/data:/var/opt/gitlab:cached'

Upgraded using:

docker-compose pull
docker-compose up -d

What is the current bug behavior?

Migration fails, the instance becomes unusable and is constantly restarted.

What is the expected correct behavior?

The migration works as smooth as in the past.

Relevant logs and/or screenshots

================================================================================
Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
================================================================================

Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '1'
---- Begin output of "bash"  "/tmp/chef-script20210823-27-1xawku" ----
STDOUT: rake aborted!
StandardError: An error has occurred, all later migrations canceled:

PG::ForeignKeyViolation: ERROR:  insert or update on table "ci_job_artifacts" violates foreign key constraint "fk_rails_c5137cb2c1_tmp"
DETAIL:  Key (job_id_convert_to_bigint)=(758) is not present in table "ci_builds".
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:292:in `block in add_concurrent_foreign_key'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:358:in `disable_statement_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:291:in `add_concurrent_foreign_key'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210809143931_finalize_job_id_conversion_to_bigint_for_ci_job_artifacts.rb:38:in `swap'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210809143931_finalize_job_id_conversion_to_bigint_for_ci_job_artifacts.rb:18:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Caused by:
ActiveRecord::InvalidForeignKey: PG::ForeignKeyViolation: ERROR:  insert or update on table "ci_job_artifacts" violates foreign key constraint "fk_rails_c5137cb2c1_tmp"
DETAIL:  Key (job_id_convert_to_bigint)=(758) is not present in table "ci_builds".
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:292:in `block in add_concurrent_foreign_key'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:358:in `disable_statement_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:291:in `add_concurrent_foreign_key'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210809143931_finalize_job_id_conversion_to_bigint_for_ci_job_artifacts.rb:38:in `swap'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210809143931_finalize_job_id_conversion_to_bigint_for_ci_job_artifacts.rb:18:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Caused by:
PG::ForeignKeyViolation: ERROR:  insert or update on table "ci_job_artifacts" violates foreign key constraint "fk_rails_c5137cb2c1_tmp"
DETAIL:  Key (job_id_convert_to_bigint)=(758) is not present in table "ci_builds".
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:292:in `block in add_concurrent_foreign_key'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:358:in `disable_statement_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:291:in `add_concurrent_foreign_key'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210809143931_finalize_job_id_conversion_to_bigint_for_ci_job_artifacts.rb:38:in `swap'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210809143931_finalize_job_id_conversion_to_bigint_for_ci_job_artifacts.rb:18:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
== 20210809143931 FinalizeJobIdConversionToBigintForCiJobArtifacts: migrating =
-- transaction_open?()
   -> 0.0000s
-- index_exists?("ci_job_artifacts", [:expire_at, :job_id_convert_to_bigint], {:name=>"index_ci_job_artifacts_on_expire_at_and_job_id_bigint", :algorithm=>:concurrently})
   -> 0.0037s
-- transaction_open?()
   -> 0.0000s
-- index_exists?("ci_job_artifacts", [:job_id_convert_to_bigint, :file_type], {:name=>"index_ci_job_artifacts_on_job_id_and_file_type_bigint", :unique=>true, :algorithm=>:concurrently})
   -> 0.0027s
-- transaction_open?()
   -> 0.0000s
-- foreign_keys("ci_job_artifacts")
   -> 0.0064s
-- execute("SET statement_timeout TO 0")
   -> 0.0001s
-- execute("ALTER TABLE ci_job_artifacts VALIDATE CONSTRAINT fk_rails_c5137cb2c1_tmp;")
-- execute("RESET statement_timeout")
   -> 0.0001s
STDERR: 
---- End output of "bash"  "/tmp/chef-script20210823-27-1xawku" ----
Ran "bash"  "/tmp/chef-script20210823-27-1xawku" returned 1

Resource Declaration:
---------------------
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb

 16:   bash "migrate #{new_resource.name} database" do
 17:     code <<-EOH
 18:     set -e
 19:     log_file="#{node['gitlab']['gitlab-rails']['log_directory']}/#{new_resource.logfile_prefix}-$(date +%Y-%m-%d-%H-%M-%S).log"
 20:     umask 077
 21:     /opt/gitlab/bin/gitlab-rake #{new_resource.rake_task} 2>& 1 | tee ${log_file}
 22:     STATUS=${PIPESTATUS[0]}
 23:     chown #{account_helper.gitlab_user}:#{account_helper.gitlab_group} ${log_file}
 24:     echo $STATUS > #{new_resource.helper.db_migrate_status_file}
 25:     exit $STATUS
 26:     EOH
 27: 
 28:     environment new_resource.environment if new_resource.property_is_set?(:environment)
 29:     new_resource.dependent_services.each do |svc|
 30:       notifies :restart, svc, :immediately
 31:     end
 32: 
 33:     not_if { new_resource.helper.migrated? }
 34:   end
 35: end

Compiled Resource:
------------------
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:16:in `block in class_from_file'

bash("migrate gitlab-rails database") do
  action [:run]
  default_guard_interpreter :default
  command nil
  backup 5
  interpreter "bash"
  declared_type :bash
  cookbook_name "gitlab"
  code "    set -e\n    log_file=\"/var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log\"\n    umask 077\n    /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}\n    STATUS=${PIPESTATUS[0]}\n    chown git:git ${log_file}\n    echo $STATUS > /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-2da7c857960\n    exit $STATUS\n"
  domain nil
  user nil
  environment {"GITLAB_ROOT_PASSWORD"=>"…removed…"}
  not_if { #code block }
end

System Info:
------------
chef_version=15.14.0
platform=ubuntu
platform_version=20.04
ruby=ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/chef-client
executable=/opt/gitlab/embedded/bin/chef-client

Results of GitLab environment info

Expand for output related to GitLab environment info 

System information
System:		
Current User:	git
Using RVM:	no
Ruby Version:	2.7.2p137
Gem Version:	3.1.4
Bundler Version:2.1.4
Rake Version:	13.0.3
Redis Version:	6.0.14
Git Version:	2.32.0
Sidekiq Version:5.2.9
Go Version:	unknown

GitLab information
Version:	14.1.3
Revision:	132256d76b1
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	12.6
URL:		https://gitlab.lan
HTTP Clone URL:	https://gitlab.lan/some-group/some-project.git
SSH Clone URL:	git@gitlab.lan:some-group/some-project.git
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: 

GitLab Shell
Version:	13.19.1
Repository storage paths:
- default: 	/var/opt/gitlab/git-data/repositories
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell
Git:		/opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 
Checking GitLab subtasks ...


Checking GitLab Shell ...


GitLab Shell: ... GitLab Shell version >= 13.19.1 ? ... OK (13.19.1)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Gitaly ...


Gitaly: ... default ... OK


Checking Gitaly ... Finished


Checking Sidekiq ...


Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1


Checking Sidekiq ... Finished


Checking Incoming Email ...


Incoming Email: ... Reply by email is disabled in config/gitlab.yml


Checking Incoming Email ... Finished


Checking LDAP ...


LDAP: ... LDAP is disabled in config/gitlab.yml


Checking LDAP ... Finished


Checking GitLab App ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
4/1 ... yes
3/2 ... yes
3/3 ... yes
6/4 ... yes
3/5 ... yes
4/6 ... yes
4/7 ... yes
9/9 ... yes
10/11 ... yes
12/12 ... yes
9/13 ... yes
Redis version >= 5.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.2)
Git version >= 2.31.0 ? ... yes (2.32.0)
Git user has default SSH configuration? ... yes
Active users: ... 5
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes


Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

Possible fixes