Handle project namespace backfilling being a blocking migration for other backfilling migrations

Context

As part of the Groups and Projects Consolidation we created a migration to backfill project namespaces that is a long running migration that runs as a batched migration. Once the project namespaces are backfilled we can use the newly backfilled project namespaces to migrate other features from projects to project namespaces.

Problem

In order to run other backfill migrations that depend on project namespaces to exist, we need to ensure first that project namespace backfill migration did run and finished first.

This is fairly straight forward on .com where we have control and can check when we want to merge the follow-up migrations, however the challenge raises for self managed instances where we do not have control over the origin project namespace backfilling migration.

So we need some way to gate-keep the follow-up backfill migrations as well as code that relies on follow-up backfilling finishing, which can be viewed as a somewhat breaking change.

NOTE: we need to cover 3 types of installs: omnibus, source, cloud native.

Possible Approach

• Add gate-keeping migrations that would finalize the original and follow-up backfilling migrations in 15.0, e.g. !82176 (diffs)
• Add clear release notes in 15.0 that original backfilling migration and follow-up migration have to be successfully completed before upgrading to 15.0 or 15.0 upgrade will fail.
  • important: if customer does not ensure all required migrations are finished before upgrading to 15.0 they may have to rollback, etc. I'm not sure what the upgrade/rollback process currently is for self managed.
• Follow-up migration code should be aware that original project namespace migration might have not finished or might have failed and should handle that gracefully, e.g. mark migration as paused and retry after a period of time.
• Code changes that rely on project namespaces backfilling should handle cases where project namespace may be missing until Gitlab 15.1 as in 15.0 the gate-keeping migrations can ensure the migration successfully finished.

Solution

Based on our zero downtime upgrade documentation, customers are directed to upgrade one release at a time, execute post-migrations and wait for background migrations to happen.

Having that we can rely on having the following upgrade path:

Project namespaces backfilling in %14.9 -> Ensure project backfilling has finished and Backfill routes namespace_id for projects in %14.10 -> followed by a finalize routes backfilling MR if needed in %15.0

Tasks

• Add a note in the Version-specific upgrading instructions. Check for example the notes for 14.2.0 and 14.3.0
• Add a release post upgrade note as well, linking to the documentation on the above task
• Maybe mention it in breaking changes if we release this in 15.0 as that is perhaps one section most customers tend to check most of the time ?