Skip to content

Draft: Reset skipped registry phase 2 imports

Steve Abrams requested to merge reset-skipped-migrations-2 into master

⚠ DO NOT MERGE ⚠ This MR is not meant to be merged but is being used as a test sandbox for reseting container repositories. Please ask a member of ~"group::package" for more info.

What does this MR do and why?

We are in the process of importing all container repositories to the new container registry. As we have worked through the container repositories, we have had to skip some imports for a variety of reasons including:

  • Trying to start the import too many times
  • Having a large number of tags (we purposely skipped these to ensure we had a stable import system before trying to import them)
  • Imports getting stuck

As we approach the end of migrating all container repositories belonging to users on the free tier, we would like to "reset" these skipped repositories so the imports can be retried.

This MR updates all container_repositories with migration_state: 'import_skipped' to migration_state: 'default'. We also reset any timestamps associated with the previous import try so there is no confusion about data from the next try.

Database

Looking at the database testing output we can see there were:

  • Total migration time of 44.6s
  • 165 UPDATE queries (~3300 records updated)
  • A few queries that had max timings outside the normal timing guidelines

Given the average query time for the UPDATE queries were ~130ms and EachBatch queries were ~43ms, my thought is the outliers are reasonable for a post-deployment migration. We could change the batch size to 10 and see a small reduction in the UPDATE timings, but I think the EachBatch timings will still have similar outliers.

We could move this to a background migration, but given the relatively small number of records being updated, I think it makes sense to keep it in a post-deployment migration.

A few other notes:

  • This migration is only taking place on GitLab.com, self-managed will be migrated separately and that work still needs to be planned, so there will be no records to update on any other instances
  • By the time this runs, we will likely have more than 3300 records to update, but we will not go over 5000 and likely stay under 4000, so the impact of adding ~50 additional UPDATEs is minimal (~10 additional seconds in migration time).

Screenshots or screen recordings

N/A

How to set up and validate locally

  1. In the rails console, create a few container repositories if you don't already have a few: 
    10.times do { FactoryBot.create(:container_repository, project: Project.last) }
  2. Update some of them to skipped: 
    ContainerRepository.limit(3).update_all(migration_state: 'import_skipped')
  3. Check the skipped imports count: 
    ContainerRepository.where(migration_state: 'import_skipped').count
  4. Run the migration
  5. Check the skipped imports count again, it should be 0

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Steve Abrams

Merge request reports