DB error upgrading 13.4.1 to 13.5.1
My GitLab instance runs on Centos and upgrades are pulled via yum upgrade
The upgrade failed when running gitlab-ctl reconfigure
, a post task, in yum. I no longer have the logs from yum upgrade
but after it failed I ran sudo gitlab-ctl reconfigure
which failed with the same error:
There was an error running gitlab-ctl reconfigure:
bash[migrate gitlab-rails database] (gitlab::database_migrations line 55) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash" "/tmp/chef-script20201025-4081-10jwq6s" ----
STDOUT: rake aborted!
StandardError: An error has occurred, all later migrations canceled:
PG::CheckViolation: ERROR: check constraint "check_d348de9e2d" is violated by some row
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1206:in `block in validate_check_constraint'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:281:in `disable_statement_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1203:in `validate_check_constraint'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1192:in `add_check_constraint'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1306:in `add_not_null_constraint'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20201005153955_add_not_null_constraint_to_compliance_project_settings.rb:11:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:59:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Caused by:
ActiveRecord::StatementInvalid: PG::CheckViolation: ERROR: check constraint "check_d348de9e2d" is violated by some row
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1206:in `block in validate_check_constraint'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:281:in `disable_statement_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1203:in `validate_check_constraint'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1192:in `add_check_constraint'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1306:in `add_not_null_constraint'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20201005153955_add_not_null_constraint_to_compliance_project_settings.rb:11:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:59:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Caused by:
PG::CheckViolation: ERROR: check constraint "check_d348de9e2d" is violated by some row
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1206:in `block in validate_check_constraint'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:281:in `disable_statement_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1203:in `validate_check_constraint'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1192:in `add_check_constraint'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1306:in `add_not_null_constraint'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20201005153955_add_not_null_constraint_to_compliance_project_settings.rb:11:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:59:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
== 20201005153955 AddNotNullConstraintToComplianceProjectSettings: migrating ==
-- current_schema()
-> 0.0003s
-- transaction_open?()
-> 0.0000s
-- current_schema()
-> 0.0002s
-- current_schema()
-> 0.0002s
-- execute("SET statement_timeout TO 0")
-> 0.0002s
-- execute("ALTER TABLE project_compliance_framework_settings VALIDATE CONSTRAINT check_d348de9e2d;")
-- execute("RESET ALL")
-> 0.0002s
STDERR:
---- End output of "bash" "/tmp/chef-script20201025-4081-10jwq6s" ----
Ran "bash" "/tmp/chef-script20201025-4081-10jwq6s" returned 1
I connected to the database to try and resolve the issue manually...
gitlabhq_production=# SELECT * FROM project_compliance_framework_settings;
project_id | framework | framework_id
------------+-----------+--------------
1 | 1 |
(1 row)
gitlabhq_production=# DELETE from project_compliance_framework_settings;
DELETE 1
Following this I ran gitlab-ctl reconfigure && gitlab-ctl restart
again and everything seems to work again. I suspect this may be a legacy of upgrading from CE to EE and back down to CE again after a few months...?
Have I missed any crucial upgrade steps given yum upgrade
finished prematurely?