After upgrading to gitlab-ce 16.4.1 reconfigure fails

After upgrading to gitlab-ce-16.4.1 from 16.3.5 using yum packages running 'gitlab-ctl reconfigure' results in the next error

There was an error running gitlab-ctl reconfigure:

rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 20) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1' ---- Begin output of "bash" ---- STDOUT: Running db:migrate rake task main: == [advisory_lock_connection] object_id: 167280, pg_backend_pid: 34727 main: == 20230823145126 SwapNotesIdToBigintForSelfManaged: migrating ================ main: -- column_exists?("notes", "id_convert_to_bigint") main: -> 0.2626s main: -- columns("notes") main: -> 0.0037s main: -- transaction_open?() main: -> 0.0000s main: -- view_exists?(:postgres_partitions) main: -> 0.0010s main: -- index_exists?("notes", :id_convert_to_bigint, {:unique=>true, :name=>"index_notes_on_id_convert_to_bigint", :algorithm=>:concurrently}) main: -> 0.0345s main: -- Index not created because it already exists (this may be due to an aborted migration or similar): table_name: notes, column_name: id_convert_to_bigint main: -- transaction_open?() main: -> 0.0000s main: -- view_exists?(:postgres_partitions) main: -> 0.0018s main: -- index_exists?("notes", [:author_id, :created_at, :id_convert_to_bigint], {:name=>:index_notes_on_author_id_created_at_id_convert_to_bigint, :algorithm=>:concurrently}) main: -> 0.0349s main: -- Index not created because it already exists (this may be due to an aborted migration or similar): table_name: notes, column_name: [:author_id, :created_at, :id_convert_to_bigint] main: -- transaction_open?() main: -> 0.0000s main: -- view_exists?(:postgres_partitions) main: -> 0.0011s main: -- index_exists?("notes", [:id_convert_to_bigint], {:where=>"confidential = true", :name=>:index_notes_on_id_convert_to_bigint_where_confidential, :algorithm=>:concurrently}) main: -> 0.0159s main: -- Index not created because it already exists (this may be due to an aborted migration or similar): table_name: notes, column_name: [:id_convert_to_bigint] main: -- transaction_open?() main: -> 0.0000s main: -- view_exists?(:postgres_partitions) main: -> 0.0017s main: -- index_exists?("notes", [:id_convert_to_bigint], {:where=>"internal = true", :name=>:index_notes_on_id_convert_to_bigint_where_internal, :algorithm=>:concurrently}) main: -> 0.0176s main: -- Index not created because it already exists (this may be due to an aborted migration or similar): table_name: notes, column_name: [:id_convert_to_bigint] main: -- transaction_open?() main: -> 0.0000s main: -- view_exists?(:postgres_partitions) main: -> 0.0015s main: -- index_exists?("notes", [:project_id, :id_convert_to_bigint], {:where=>"NOT system", :name=>:index_notes_on_project_id_id_convert_to_bigint_system_false, :algorithm=>:concurrently}) main: -> 0.0154s main: -- Index not created because it already exists (this may be due to an aborted migration or similar): table_name: notes, column_name: [:project_id, :id_convert_to_bigint] main: -- transaction_open?() main: -> 0.0000s main: -- view_exists?(:postgres_partitions) main: -> 0.0011s main: -- index_exists?("notes", [:id_convert_to_bigint, :noteable_type], {:where=>"note ~~ '%@%'::text", :name=>:note_mentions_temp_index_convert_to_bigint, :algorithm=>:concurrently}) main: -> 0.0169s main: -- Index not created because it already exists (this may be due to an aborted migration or similar): table_name: notes, column_name: [:id_convert_to_bigint, :noteable_type] main: -- transaction_open?() main: -> 0.0000s main: -- execute("ALTER TABLE notes RENAME COLUMN id TO id_tmp") main: -> 0.0037s main: -- execute("ALTER TABLE notes RENAME COLUMN id_convert_to_bigint TO id") main: -> 0.0005s main: -- execute("ALTER TABLE notes RENAME COLUMN id_tmp TO id_convert_to_bigint") main: -> 0.0004s main: -- quote_table_name("trigger_080e73845bfd") main: -> 0.0000s main: -- execute("ALTER FUNCTION "trigger_080e73845bfd" RESET ALL") main: -> 0.0005s main: -- execute("ALTER SEQUENCE notes_id_seq OWNED BY notes.id") main: -> 0.0037s main: -- change_column_default("notes", :id, #<Proc:0x00007f9aa76d99a0 /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20230823145126_swap_notes_id_to_bigint_for_self_managed.rb:106 (lambda)>) main: -> 0.0172s main: -- change_column_default("notes", :id_convert_to_bigint, 0) main: -> 0.0330s main: -- execute("ALTER TABLE notes DROP CONSTRAINT notes_pkey") rake aborted! StandardError: An error has occurred, all later migrations canceled:

PG::DependentObjectsStillExist: ERROR: cannot drop constraint notes_pkey on table notes because other objects depend on it DETAIL: constraint fk_c262d728d4 on table system_note_metadata depends on index notes_pkey HINT: Use DROP ... CASCADE to drop the dependent objects too. /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20230823145126_swap_notes_id_to_bigint_for_self_managed.rb:111:in block in swap' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in run_block' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in block in run_block_with_lock_timeout' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in run_block_with_lock_timeout' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in run' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retries_helpers.rb:52:in with_lock_retries' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/v2.rb:99:in with_lock_retries' /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20230823145126_swap_notes_id_to_bigint_for_self_managed.rb:92:in swap' /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20230823145126_swap_notes_id_to_bigint_for_self_managed.rb:70:in up' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in block in exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:37:in within' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in ddl_transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in execute_migration_in_transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in block in with_advisory_lock_connection' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in with_advisory_lock_connection' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:123:in configure_database' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:101:in block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:25:in load' /opt/gitlab/embedded/bin/bundle:25:in `'

Caused by: ActiveRecord::StatementInvalid: PG::DependentObjectsStillExist: ERROR: cannot drop constraint notes_pkey on table notes because other objects depend on it DETAIL: constraint fk_c262d728d4 on table system_note_metadata depends on index notes_pkey HINT: Use DROP ... CASCADE to drop the dependent objects too. /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20230823145126_swap_notes_id_to_bigint_for_self_managed.rb:111:in block in swap' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in run_block' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in block in run_block_with_lock_timeout' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in run_block_with_lock_timeout' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in run' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retries_helpers.rb:52:in with_lock_retries' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/v2.rb:99:in with_lock_retries' /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20230823145126_swap_notes_id_to_bigint_for_self_managed.rb:92:in swap' /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20230823145126_swap_notes_id_to_bigint_for_self_managed.rb:70:in up' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in block in exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:37:in within' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in ddl_transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in execute_migration_in_transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in block in with_advisory_lock_connection' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in with_advisory_lock_connection' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:123:in configure_database' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:101:in block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:25:in load' /opt/gitlab/embedded/bin/bundle:25:in `'

Caused by: PG::DependentObjectsStillExist: ERROR: cannot drop constraint notes_pkey on table notes because other objects depend on it DETAIL: constraint fk_c262d728d4 on table system_note_metadata depends on index notes_pkey HINT: Use DROP ... CASCADE to drop the dependent objects too. /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20230823145126_swap_notes_id_to_bigint_for_self_managed.rb:111:in block in swap' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in run_block' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in block in run_block_with_lock_timeout' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in run_block_with_lock_timeout' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in run' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retries_helpers.rb:52:in with_lock_retries' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/v2.rb:99:in with_lock_retries' /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20230823145126_swap_notes_id_to_bigint_for_self_managed.rb:92:in swap' /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20230823145126_swap_notes_id_to_bigint_for_self_managed.rb:70:in up' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in block in exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:37:in within' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in ddl_transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in execute_migration_in_transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in block in with_advisory_lock_connection' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in with_advisory_lock_connection' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:123:in configure_database' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:101:in block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:25:in load' /opt/gitlab/embedded/bin/bundle:25:in `' Tasks: TOP => db:migrate (See full trace by running task with --trace) STDERR: ---- End output of "bash" ---- Ran "bash" returned 1