Migration failed when updating from 15.4.6 CE to 15.10.2CE

I am getting the following error when I am trying to upgrade my instance from 15.4.6 CE to 15.10.2CE. I have already tried the following things

  • Manually migrating the failed migration didn't work
  • Tried to restore to older version from the backup and still getting the same error when running sudo gitlab-ctl reconfigure

Please find the trace below

[2023-04-12T20:32:21+00:00] ERROR: Running exception handlers There was an error running gitlab-ctl reconfigure:

rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 18) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1' ---- Begin output of "bash" ---- STDOUT: rake aborted! StandardError: An error has occurred, all later migrations canceled:

PG::CheckViolation: ERROR: check constraint "check_70f294ef54" is violated by some row /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/constraints_helpers.rb:119:in block in validate_check_constraint' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/timeout_helpers.rb:31:in disable_statement_timeout' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/constraints_helpers.rb:116:in validate_check_constraint' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/constraints_helpers.rb:232:in validate_not_null_constraint' /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20230223014251_validate_not_null_constraint_on_oauth_access_tokens_expires_in.rb:7:in up' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in block in exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:37:in within' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:19:in exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in ddl_transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:115:in configure_database' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:95:in block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in '

Caused by: ActiveRecord::StatementInvalid: PG::CheckViolation: ERROR: check constraint "check_70f294ef54" is violated by some row /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/constraints_helpers.rb:119:in block in validate_check_constraint' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/timeout_helpers.rb:31:in disable_statement_timeout' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/constraints_helpers.rb:116:in validate_check_constraint' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/constraints_helpers.rb:232:in validate_not_null_constraint' /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20230223014251_validate_not_null_constraint_on_oauth_access_tokens_expires_in.rb:7:in up' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in block in exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:37:in within' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:19:in exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in ddl_transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:115:in configure_database' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:95:in block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in '

Caused by: PG::CheckViolation: ERROR: check constraint "check_70f294ef54" is violated by some row /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/constraints_helpers.rb:119:in block in validate_check_constraint' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/timeout_helpers.rb:31:in disable_statement_timeout' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/constraints_helpers.rb:116:in validate_check_constraint' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/constraints_helpers.rb:232:in validate_not_null_constraint' /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20230223014251_validate_not_null_constraint_on_oauth_access_tokens_expires_in.rb:7:in up' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in block in exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:37:in within' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:19:in exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in ddl_transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:115:in configure_database' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:95:in block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in ' Tasks: TOP => db:migrate (See full trace by running task with --trace) main: == 20230223014251 ValidateNotNullConstraintOnOauthAccessTokensExpiresIn: migrating main: -- execute("SET statement_timeout TO 0") main: -> 0.0005s main: -- execute("ALTER TABLE oauth_access_tokens VALIDATE CONSTRAINT check_70f294ef54;") main: -- execute("RESET statement_timeout") main: -> 0.0004s STDERR: ---- End output of "bash" ---- Ran "bash" returned 1