## Context This is part of the work to upgrade and migrate the GitLab.com container registry to a new version backed by a metadata database and online garbage collection (https://gitlab.com/groups/gitlab-org/-/epics/5523). This will be achieved following the gradual migration plan detailed in https://gitlab.com/gitlab-org/container-registry/-/issues/374. This specific epic is meant to collect the tasks required for testing [Phase 2](https://gitlab.com/gitlab-org/container-registry/-/issues/374#phase-2-migrate-existing-repositories) of the migration plan in staging. ## Plan | Step | Category | Task | Description | Risks | Required Teams | Status | | ---- | ----------------- | ------------------| ------------------------ | -------------------- | ----------------- | ---------------| | 1 | Setup | Inventory | Run the registry CLI `inventory` command against the staging bucket and dump the output in a spreadsheet. We can then sort and filter existing repositories by namespace and tag count. To ID namespaces and repositories that are good testing candidates | Moderately increased workload to staging registry bucket. | Delivery, Package | :white_check_mark: The result is [here](https://docs.google.com/spreadsheets/d/1pIOS2o6AGiZ1Hqr3Mlr3YY2FF1sXPp-KCJwY6lWcwXo/edit?usp=sharing) | | 2| Setup | Configure Registry Import API Options Under the `Migration` Stanza |Configuring options under `importnotification` | None. | Delivery, Package | :white_check_mark: | | 3| Setup | Protect Namespaces | Protect a few top-level namespaces using the `container_registry_phase_2_deny_list` | None. | Delivery, Package | :white_check_mark: | | 4 | Setup | Configure Migration Workers on Rails| Ensure workers are set up and configured properly and ready to import. | None. | Delivery, Package | :white_check_mark: | | 5 | Imports | Simple repository import | Import a repository with only a couple of images and pause the migration workers in order to manually confirm the data integrity of the imported repository. | Temporary write disruption to repositories. | Package | :white_check_mark: | | 6 | Imports | Limited Import | Configure the migration workers to only import repositories with < 100 tags and moderate rate via the `container_registry_migration_phase2_enqueue_speed_X` feature flag, allow them to import for a couple of hours and then and pause the migration workers in order to manually confirm the data integrity of a sample of imported images. Additionally, ensure that the workers queue repositories as expected. | Moderately increased workload to staging registry app, bucket, and database. Temporary write disruption to repositories. | Package | :white_check_mark: | | 7 | Load testing | Large Repository Import | Import a repository with around 500 images. Work with Delivery to ensure that performance during this import remains with acceptable bounds. | Potentially significantly increased workload to staging registry app, bucket, and database. Temporary write disruption to repository. | Delivery, Package | :white_check_mark: | | 8 | Load testing | Write Contention | Import a repository while generating new incoming writes. Ensure writes succeed during pre import, and are blocked during import. | Moderately increased workload to staging registry app, bucket, and database. | Delivery, Package | :white_check_mark: | | 9 | Failure scenarios | Notification Failure | Configure the registry not to send import notifications and pre import a test repository. The guard worker will ask for the migration status on the Container Registry. In this case, the guard worker will abort the migration so that the enqueuer can pick it up in priority and retry it. During that retry, the enqueuer should check pre import has been done and should transition the import to the right state — pre import completed. Observe that import completes and that self-healing happens again, resulting in rails marking the repository as imported. | None. | Delivery, Package | :white_check_mark: | | 10 | Failure scenarios | Pre Import Timeout | Configure the registry to pause for an extended period of time after imports, and also configure the registry to timeout on imports well before that. Import a repository and ensure that registry reports the timeout as expected. Ensure that workers handle the timeout notification as expected. | None. | Package | :white_check_mark: | | 11 | Imports | Long Running Import | Configure the registry to pause for an hour after imports. Ensure that workers log the long-running import as expected. | None. | Package | :white_check_mark: | |12| Imports | Migrate Historic Staging Repositories | Configure import workers to import remaining repositories on staging. | Potentially reduced availability as repositories import. Increased load during import, permanently increased load on staging database.| Package | :white_check_mark: | |13| Imports | Import Protected Namespaces | Begin importing the top-level namespaces protected by the `container_registry_migration_phase2_deny_list` one at a time. | None. | Package | :white_check_mark: | During all these tests we should: - Check that logs, metrics, and error reports match expectations and are sufficient to debug any issues; - Debug any errors that may pop up, taking special attention to rule out data consistency issues. ## Questions - How can we rollback migrated repositories to use the old code path and not the metadata DB in case of problems? Would we need to delete them from the metadata DB or are there better ways? - We can use feature flags to change the code path. - Do we expect QA tests to fail if they try to do registry write requests for repos that are locked for writes because of being migrated? (I hope we always create new repos for QA tests so that there is nothing to be migrated during QA.) - We use docker, which has a 45sec retry window. - Do we still mirror metadata on prod? - Yes - How do we test the code path? - We check the database, if we find it on the database, we check it’s migration status, then we know if it is native or migrated, if so it takes the new path, if not, it takes the old path (we check filesystem to be sure). If the import is in progress we cancel the import and go with the old path. ## Status All tests and followups completed successfully.