Staging Background Migration Marked as failed due to rollback

The container registry first experimental background migration on staging was safely marked as failed because the new registry version v4.23.0 that references the migration function was rolled back to v4.22.0 that does not reference the migration function for the background migration in the database.

As a result, the older registry background migration workers attempted the migration without the updated code that references the necessary function, and then marked the migration as failed with the error: work function not found.

Next Steps:

  1. Documentation Update: Add documentation to ensure that background migration entries are only added after a GitLab release that includes the registry with the corresponding migration function. This is the MVC here, we can iterate on this, see: Revisit Retry Behavior for Background Migration... (#1616 - closed)

  2. Restart the Failed Migration on Staging: Reset the failure to restart the background migration on staging once the environment has been updated with the new pods containing the required function.

UPDATE batched_background_migrations
SET failure_error_code = NULL,
status = 1 -- 'active'
WHERE name = 'copy_manifests_media_type_id_column_to_media_type_id_convert_to_bigint_column';

Note: This does not affect production as background migrations are disabled there. The manual intervention is needed because ChatOps support is not developed yet.

Edited by SAhmed