Error migrating from 14.0.12 to 14.8.2
Sumary
Error migrating from 14.0.12 to 14.8.2: batched_background_migration_job_transition_logs" does not exist
There was an error running gitlab-ctl reconfigure:
rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash" "/tmp/chef-script20220228-16729-1d1rc1x" ----
STDOUT: rake aborted!
StandardError: An error has occurred, all later migrations canceled:
Expected batched background migration for the given configuration to be marked as 'finished', but it is 'finalizing': {:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"ci_builds", :column_name=>"id", :job_arguments=>[["id", "stage_id"], ["id_convert_to_bigint", "stage_id_convert_to_bigint"]]}
Finalize it manualy by running
sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_builds,id,'[["id"\, "stage_id"]\, ["id_convert_to_bigint"\, "stage_id_convert_to_bigint"]]']
For more information, check the documentation
https://docs.gitlab.com/ee/user/admin_area/monitoring/background_migrations.html#database-migrations-failing-because-of-batched-background-migration-not-finished
When trying to run, what it specifies I get :
sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_builds,id,'[["id"\, "stage_id"]\, ["id_convert_to_bigint"\, "stage_id_convert_to_bigint"]]']
rake aborted!
ActiveRecord::StatementInvalid: PG::UndefinedTable: ERROR: relation "batched_background_migration_job_transition_logs" does not exist
LINE 8: WHERE a.attrelid = '"batched_background_migration_job_trans...
^
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/schema_cache_with_renamed_table.rb:25:in `columns'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/schema_cache_with_renamed_table.rb:29:in `columns_hash'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_job.rb:59:in `block (2 levels) in '
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:112:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:172:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:110:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:125:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:77:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:285:in `block in transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:284:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_wrapper.rb:27:in `rescue in perform'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_wrapper.rb:26:in `perform'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:70:in `block in finalize'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:70:in `finalize'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:10:in `finalize'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/background_migrations.rake:14:in `block (3 levels) in '
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `'
Caused by:
PG::UndefinedTable: ERROR: relation "batched_background_migration_job_transition_logs" does not exist
If I run commands to check how many migrations are left:
gitlab-rails runner -e production 'puts Gitlab::Database::BackgroundMigrationJob.pending.count' 0 gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining' 0
If I run query in DB:
sudo gitlab-psql
psql (12.7, server 12.6)
Type "help" for help.
gitlabhq_production=# select job_class_name, table_name, column_name, job_arguments from batched_background_migrations where status <> 3;
job_class_name | table_name | column_name | job_arguments
---------------------------------------+--------------------+-------------+------------------------------------
CopyColumnUsingBackgroundMigrationJob | ci_stages | id | [["id"], ["id_convert_to_bigint"]]
CopyColumnUsingBackgroundMigrationJob | ci_builds_metadata | id | [["id"], ["id_convert_to_bigint"]]
(2 rows)
Edited by Luka