Error in gitlab-ctl reconfigure after upgrade to 16.10.2-ee from 16.9.0-ee
Hi everyone
Summary
So I tried to update my self-managed gitlab instance from 16.9.0 to 16.10.2. But during the gitlab-ctl reconfigure step it crashes at the gitlab rails migration task.
Steps to reproduce
Having gitlab 16.9.0-ee sudo apt update && sudo apt install gitlab-ee
What is the current bug behavior?
The gitlab reconfigure step crashes and the upgrade is cancelled. It crashed during rails migration task due to violation of foreign key in the "raw_usage_data" table.
I already tried as seen in other posts deleting the fk but doesn't work so I reloaded my VM to pre upgrade snapshot.
What is the expected correct behavior?
Gitlab should be updated
Relevant logs
Here you can see the output given during upgrade crash.
Relevant logs
main: == [advisory_lock_connection] object_id: 53340, pg_backend_pid: 32070rake aborted! StandardError: An error has occurred, all later migrations canceled:PG::ForeignKeyViolation: ERROR: insert or update on table "raw_usage_data" violates foreign key constraint "fk_8e21125854" DETAIL: Key (organization_id)=(1) is not present in table "organizations". /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:290:in
block in add_concurrent_foreign_key' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/timeout_helpers.rb:31:in
disable_statement_timeout' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:289:inadd_concurrent_foreign_key' /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20240221152300_add_organisation_to_raw_usage_data.rb:13:in
up' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:inblock in exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:40:in
within' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:inexec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in
exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:54:inddl_transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in
execute_migration_in_transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:15:inblock in with_advisory_lock_connection' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in
with_advisory_lock_connection' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:136:inconfigure_database' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:107:in
configure_pg_databases' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:94:inblock (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:25:in
load' /opt/gitlab/embedded/bin/bundle:25:in `'Caused by: ActiveRecord::InvalidForeignKey: PG::ForeignKeyViolation: ERROR: insert or update on table "raw_usage_data" violates foreign key constraint "fk_8e21125854" DETAIL: Key (organization_id)=(1) is not present in table "organizations". /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:290:in
block in add_concurrent_foreign_key' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/timeout_helpers.rb:31:in
disable_statement_timeout' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:289:inadd_concurrent_foreign_key' /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20240221152300_add_organisation_to_raw_usage_data.rb:13:in
up' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:inblock in exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:40:in
within' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:inexec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in
exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:54:inddl_transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in
execute_migration_in_transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:15:inblock in with_advisory_lock_connection' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in
with_advisory_lock_connection' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:136:inconfigure_database' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:107:in
configure_pg_databases' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:94:inblock (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:25:in
load' /opt/gitlab/embedded/bin/bundle:25:in `'Caused by: PG::ForeignKeyViolation: ERROR: insert or update on table "raw_usage_data" violates foreign key constraint "fk_8e21125854" DETAIL: Key (organization_id)=(1) is not present in table "organizations". /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:290:in
block in add_concurrent_foreign_key' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/timeout_helpers.rb:31:in
disable_statement_timeout' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:289:inadd_concurrent_foreign_key' /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20240221152300_add_organisation_to_raw_usage_data.rb:13:in
up' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:inblock in exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:40:in
within' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:inexec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in
exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:54:inddl_transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in
execute_migration_in_transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:15:inblock in with_advisory_lock_connection' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in
with_advisory_lock_connection' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:136:inconfigure_database' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:107:in
configure_pg_databases' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:94:inblock (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:25:in
load' /opt/gitlab/embedded/bin/bundle:25:in `' Tasks: TOP => db:migrate (See full trace by running task with --trace) STDERR: ---- End output of "bash" ---- Ran "bash" returned 1There was an error running gitlab-ctl reconfigure. Please check the output above for more details.
Details of package version
Provide the package version installation details
* Operating System: `Debian 11` * Installation Target, : * VM:`V-Sphere`
- Installation Type, :
- Upgrade from version
16.9.0
- Upgrade from version
- Is there any other software running on the machine:
No
- Is this a single or multiple node installation? : Single
Thanks for your help.