## Context We're performing an upgrade/migration of the GitLab.com Container Registry to a new platform backed by a metadata database ([gitlab-org&5523](https://gitlab.com/groups/gitlab-org/-/epics/5523)). This process is split into two phases: - **Phase 1**: Route newly created container repositories to the new platform. - **Phase 2**: Import existing container repositories to the new platform. For [Phase 2](https://gitlab.com/groups/gitlab-org/-/epics/6427), we'll be driving the import of existing repositories from the Rails side. This issue details the requirements to implement the logic required for this. It's highly recommended to read through the [gradual migration plan](https://gitlab.com/gitlab-org/container-registry/-/issues/374) for a detailed explanation of the upgrade/migration strategy. Here we'll only provide the minimum required information to understand the purpose and requirements of this issue. We'll provide links to relevant sections of the plan whenever appropriate. ## Overview The `container_repositories` table in the Rails database will serve as inventory for Phase 2. Once [Phase 1](https://gitlab.com/groups/gitlab-org/-/epics/6426) is completed, say on the date `T`, we know that all `container_repositories` whose `created_at` is less than `T` will need to be imported into the new platform during Phase 2. For the remaining of this issue, we'll call such container repositories "eligible". On the Container Registry side, the import of repositories is split into two steps: - **Pre-import**: All data within a repository is imported into the new platform, except tags; - **Import**: Any data created since the pre-import will be imported during this step. Additionally, all tags are imported. The target container repository must be locked against writes for the duration of this step. You can read more about these steps [here](https://gitlab.com/gitlab-org/container-registry/-/issues/374#two-pass-approach). The idea is then to loop over the eligible container repositories on the Rails side and do the following (high-level): 1. Trigger pre-import for container repository `R` by calling a new endpoint on the Container Registry API 1. Once the pre-import for `R` is complete, lock `R` against writes 1. Trigger import by calling a second new endpoint on the Container Registry API 1. Unlock `R` once the import completes. ## Requirements - Feature flag to start and stop the processing of eligible repositories. When stopping, the intention is not to abort inflight imports but rather to avoid additional repositories from being picked up; - Process repositories in the background using workers; - Feature flag to control the number of workers processing repositories in "parallel". Start with a single worker but make it possible to enable parallelism; - Keep track of the migration status of each repository using a state machine (new `import_status` column on the `container_repositories` table); - Trigger (pre-)imports by calling a new endpoint on the Container Registry API; - Expose a new API endpoint that the Container Registry can call to notify Rails about the completion of a (pre-)import; - An import should only be triggered after the pre-import completes successfully; - An import should be triggered ASAP once the corresponding pre-import completes successfully; - If a repository is being imported (not pre-imported), refuse to serve JWT tokens (`/jwt/auth` endpoint) with `push` or `delete` scopes for that repository. This applies to both external (users) and internal (Rails) operations. This is the locking against writes; - Cap duration of imports (not pre-imports) with a configurable duration, avoiding locking repositories against writes for too long; - Automatically retry failed or aborted (pre-)imports up to 3 times. Repositories with more than three failed imports should be skipped (we'll deal with them later); - Skip repositories that have more than a configurable number of tags. The intention is to start by importing small repositories (with up to 100 tags) and increase this limit as we gain confidence; - Process repositories plan-by-plan. The intention is to start with repositories on the free plan and then move up; - Feature flag to act as deny list of top-level namespaces. Repositories that belong to top-level namespaces who have this feature flag enabled should be skipped. The intention is to protect major customers during the initial rollout; - Feature flag to control the percentage of eligible (existing and not belonging to a protected top-level namespace) repositories for which we'll trigger a pre-import (and the subsequent import); - If not hearing back from the Container Registry in e.g., 30 minutes since the (pre-)import started, poll for the status of the (pre-)import by calling a new endpoint on the Container Registry API. The intention is to protect against lost updates; The new Container Registry API is documented [here](https://gitlab.com/gitlab-org/container-registry/-/blob/master/docs-gitlab/api.md#import-repository). ## Cross-team collaboration - ~"team::Scalability": @qmnguyen0711 is the Scalability DRI for Phase 2 and will be assisting us during the design and implementation of this.