Upgrade error `PG::UndefinedObject: ERROR: constraint "fk_262d4c2d19" for table "ci_pipelines" does not exist`
Hi,
I'm upgrading a GitLab v16.5.1 to v16.6.1 with gitlab-ee package on Ubuntu 22.04 and I'm getting the following error PG::UndefinedObject: ERROR: constraint "fk_262d4c2d19" for table "ci_pipelines" does not exist:
# apt-get -V dist-upgrade
[...]
main: -- execute("ALTER TABLE \"ci_pipelines\"\nRENAME CONSTRAINT \"fk_262d4c2d19\" TO \"temp_name_for_renaming\"\n")
rake aborted!
StandardError: An error has occurred, all later migrations canceled:
PG::UndefinedObject: ERROR: constraint "fk_262d4c2d19" for table "ci_pipelines" does not exist
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/constraints_helpers.rb:253:in `rename_constraint'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/swapping.rb:30:in `swap_foreign_keys'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20231025025733_swap_columns_for_ci_pipelines_pipeline_id_bigint_for_self_host.rb:48:in `block in swap'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in `run_block'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in `block in run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in `run'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retries_helpers.rb:52:in `with_lock_retries'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/v2.rb:99:in `with_lock_retries'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20231025025733_swap_columns_for_ci_pipelines_pipeline_id_bigint_for_self_host.rb:37:in `swap'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20231025025733_swap_columns_for_ci_pipelines_pipeline_id_bigint_for_self_host.rb:21:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:37:in `within'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in `block in with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in `with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:126:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:101:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Caused by:
ActiveRecord::StatementInvalid: PG::UndefinedObject: ERROR: constraint "fk_262d4c2d19" for table "ci_pipelines" does not exist
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/constraints_helpers.rb:253:in `rename_constraint'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/swapping.rb:30:in `swap_foreign_keys'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20231025025733_swap_columns_for_ci_pipelines_pipeline_id_bigint_for_self_host.rb:48:in `block in swap'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in `run_block'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in `block in run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in `run'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retries_helpers.rb:52:in `with_lock_retries'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/v2.rb:99:in `with_lock_retries'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20231025025733_swap_columns_for_ci_pipelines_pipeline_id_bigint_for_self_host.rb:37:in `swap'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20231025025733_swap_columns_for_ci_pipelines_pipeline_id_bigint_for_self_host.rb:21:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:37:in `within'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in `block in with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in `with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:126:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:101:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Caused by:
PG::UndefinedObject: ERROR: constraint "fk_262d4c2d19" for table "ci_pipelines" does not exist
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/constraints_helpers.rb:253:in `rename_constraint'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/swapping.rb:30:in `swap_foreign_keys'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20231025025733_swap_columns_for_ci_pipelines_pipeline_id_bigint_for_self_host.rb:48:in `block in swap'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in `run_block'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in `block in run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in `run'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retries_helpers.rb:52:in `with_lock_retries'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/v2.rb:99:in `with_lock_retries'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20231025025733_swap_columns_for_ci_pipelines_pipeline_id_bigint_for_self_host.rb:37:in `swap'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20231025025733_swap_columns_for_ci_pipelines_pipeline_id_bigint_for_self_host.rb:21:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:37:in `within'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in `block in with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in `with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:126:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:101:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
STDERR:
---- End output of "bash" ----
Ran "bash" returned 1
===
There was an error running gitlab-ctl reconfigure. Please check the output above for more
details.
===
dpkg: erreur de traitement du paquet gitlab-ee (--configure) :
installed gitlab-ee package post-installation script subprocess returned error exit status 1
Des erreurs ont été rencontrées pendant l'exécution :
gitlab-ee
needrestart is being skipped since dpkg has failed
E: Sub-process /usr/bin/dpkg returned an error code (1)I apparently resolved this error by manually applying the constraint fk_262d4c2d19 I found on this issue #430722 (closed) and replaying the upgrade:
ALTER TABLE "ci_pipelines" ADD CONSTRAINT "fk_262d4c2d19" FOREIGN KEY (auto_canceled_by_id) REFERENCES ci_pipelines(id) ON DELETE SET NULL;Edited by Jérôme Augé