Background Migrations Failed job BackfillInternalOnNotes

Here is my forum post on the issue https://forum.gitlab.com/t/background-migrations-failed-job-backfillinternalonnotes/98851

The issue is that i have a failed Migration BackfillInternalOnNotes which is stopping me from upgrading to 16.9.2 currently on 16.7.7 if i run sudo gitlab-rake gitlab:elastic:list_pending_migrations

BackfillHiddenOnIssues [Obsolete]
BackfillHashedRootNamespaceIdOnNotes
AddHashedRootNamespaceIdToMergeRequests
AddHashedRootNamespaceIdToIssues
AddHashedRootNamespaceIdToCommits
BackfillHashedRootNamespaceIdOnIssues
BackfillHashedRootNamespaceIdOnMergeRequests
BackfillHashedRootNamespaceIdToCommits
BackfillWikiPermissionsInMainIndex
MigrateWikisToSeparateIndex
AddHiddenToMergeRequests
BackfillHiddenOnMergeRequests
AddSuffixProjectInWikiRid
BackfillProjectPermissionsInBlobsUsingPermutations
BackfillProjectPermissionsInBlobs
AddCiCatalogToProject
ReindexWikisToFixPermissionsAndTraversalIds
MigrateProjectsToSeparateIndex
BackfillMilestonePermissionsToMilestoneDocuments
AddSchemaVersionToCommits
CreateEpicIndex
AddArchivedToIssues
BackfillArchivedOnIssues
BackfillExistingGroupWiki
ReindexCommitsToFixPermissions
AddArchivedToMergeRequests
BackfillArchivedOnMergeRequests
BackfillArchivedOnWorkItems
AddArchivedToCommits
AddArchivedToNotes
AddArchivedToMainIndex
AddArchivedToWikis
ReindexWikisToFixRouting
BackfillArchivedOnNotes
BackfillEpics
BackfillArchivedFieldInCommits
RemoveWikisFromMainIndex
RemoveProjectsFromMainIndex
ReindexWorkItemsToFixArchived
BackfillArchivedFieldInBlob
AddSchemaVersionToMergeRequest
AddSchemaVersionToNote
BackfillArchivedOnMilestones
ForceReindexCommitsFromMainIndex
ReindexIssuesToFixLabelIds
ReindexWorkItemToFixLabelIds
RemoveLeftoverCommitsFromMainIndex
RemoveHashedRootNamespaceIdFromUsers
ReindexAndRemoveLeftoverNotesFromMainIndex
ReindexAndRemoveLeftoverMergeRequestInMainIndex
ReindexEpicsToFixLabelIds
ReindexWikisToFixRoutingAndBackfillArchived
ReindexIssuesToUpdateAnalyzer

When upgrading to 16.9.2 i get the error PG::UndefinedTable: ERROR: relation "gitlab_partitions_static.issue_search_data_00_search_vector_idx" does not exist

failed Background Migration BackfillInternalOnNotes

Output of gitlab-rake gitlab:background_migrations:finalize[BackfillInternalOnNotes,notes,id,'[]']

rake aborted!
ActiveRecord::StatementInvalid: PG::CheckViolation: ERROR:  no partition of relation "batched_background_migration_job_transition_logs" found for row
DETAIL:  Partition key of the failing row contains (created_at) = (2024-01-25 00:34:50.123161+00).
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:61:in `block (2 levels) in <class:ConnectionProxy>'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:359:in `block in transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:358:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_job.rb:84:in `block (2 levels) in <class:BatchedJob>'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:359:in `block in transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:358:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_wrapper.rb:34:in `rescue in perform'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_wrapper.rb:28:in `perform'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:30:in `run_migration_job'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:140:in `run_migration_while'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:80:in `finalize'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:10:in `finalize'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/background_migrations.rake:72:in `finalize_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/background_migrations.rake:18:in `block (3 levels) in <top (required)>'

System information
System:         Ubuntu 20.04
Proxy:          no
Current User:   git
Using RVM:      no
Ruby Version:   3.1.4p223
Gem Version:    3.4.22
Bundler Version:2.4.22
Rake Version:   13.0.6
Redis Version:  7.0.15
Sidekiq Version:6.5.12
Go Version:     unknown

GitLab information
Version:        16.7.7-ee
Revision:       8d2bc7e01af
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     13.13
URL:            https://mydomain:8050
HTTP Clone URL: https://mydomain:8050/some-group/some-project.git
SSH Clone URL:  git@mydomain:some-group/some-project.git
Elasticsearch:  no
Geo:            no
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers:

GitLab Shell
Version:        14.32.0
Repository storages:
- default:      unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell

Gitaly
- default Address:      unix:/var/opt/gitlab/gitaly/gitaly.socket
- default Version:      16.7.7
- default Git Version:  2.42.0

What should i try to fix this?

Edited Mar 18, 2024 by ageekhere