Add documentation about batched background migrations not being able to run when the cron job is not enabled
We have not enabled the cron job for BatchedBackgroundMigrationWorker
on self-managed instances in %14.0, which results to the error mentioned in gitlab-org/charts/gitlab#2785 (closed) when an instance upgrades to a higher (minor) version that includes a column swap migration (e.g. !64577 (merged))
Our plan to resolve this is the following:
-
Enable the cron job for
BatchedBackgroundMigrationWorker
on self-managed instances and backport the change to %14.0 as well (#334996 (closed)) -
Revert the column swap migration for
push_event_payloads
(!64577 (merged)) on .com and do not release the migration to self managed instances. -
Release all the column swap changes on %14.2 onwards
That makes %14.1 a safe release, with the batched background migrations able to automatically run and finish the backfilling of all
bigint
columns. -
Require that all self-managed instance upgrade through %14.1
That will allow instances to catchup, finish all the pending migrations and be ready for the column swap once they upgrade to the next version they plan to upgrade
The steps above will guarantee us that we have no instances facing the error mentioned in gitlab-org/charts/gitlab#2785 (closed), but there is no way that we can enforce that requirement right now, so instances may end up skipping %14.1 and directly upgrading to a higher minor or major (%15.0) version. If they are upgrading from a version prior to the patch that introduces the cron job in %14.0 as well, then they'll get the error mentioned in gitlab-org/charts/gitlab#2785 (closed) once more.
For that reason, we also have to add documentation on how to recover from that error, either by running the migrations manually or going back to the 14.0.Z
patch, letting the migrations run and then upgrade.
In this is issue we want to cover all the documentation related updates:
-
Require that all self-managed instance upgrade through %14.1
-
Add a Version-specific upgrading instruction similar to the requirement for going through 13.1 or 12.0.Z
-
Explain that an alternative path is to go through
14.0.Z
, where Z is the patch version that !65106 (merged) will end up being included to. -
Update release notes for %14.1 with that information as well (will be covered in #335005 (closed) to unblock this issue from completing)
-
-
Update the error message that the migrations show when the batched background migrations are still active:
Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active' ..
To also include a link to the Batched Background Migrations documentation
-
Add a section on the Batched Background Migrations documentation that will explain the steps necessary when upgrading from
14.0.Z
- to any release other than %14.1 and they see that error.