Explore alternative options for the new Container Registry deployment/migration setup
## Context
The Package team has been working on a new version of the Container Registry that relies on a [metadata database](https://docs.gitlab.com/ee/architecture/blueprints/container_registry_metadata_database/) to enable online garbage collection and unblock the implementation of several other features.
With the new version now implemented, we've been working on the rollout, starting with GitLab.com (https://gitlab.com/groups/gitlab-org/-/epics/5523). We have proposed a [gradual deployment/migration plan for GitLab.com](https://gitlab.com/gitlab-org/container-registry/-/issues/191) to enable this.
## Status
A final alternative approach was detailed in https://gitlab.com/gitlab-org/container-registry/-/issues/374.
## Concerns
Some concerns have been raised regarding the original deployment/migration strategy linked above. These concerns are related not only to GitLab.com but also to self-managed customers that may not be able to afford a few hours of read-only mode required to perform a one-time [offline migration](https://gitlab.com/gitlab-org/container-registry/-/blob/master/docs-gitlab/database-import-tool.md#importing-existing-data-to-the-database):
1. **Separate bucket:** The original plan proposes using a separate bucket for the new registry. This requires additional deployment and maintenance work;
1. **Separate instance/cluster:** The original plan proposes using a separate registry instance/cluster, operating side-by-side with the existing one during the whole migration. This requires additional deployment and maintenance work;
1. **Separate migration service:** The original plan proposes using a separate service to handle the migration of existing repositories from the old registry to the new one. This requires additional deployment and maintenance work;
1. **Migration speed:** The need for an extended period for the data migration has been highlighted as a concern.
## Proposal
Investigate possible alternatives to simplify the deployment/migration plan and mitigate the concerns described above.
## Status
We had a few followup meetings with `@ayufan` and `@stanhu` to discuss possible alternatives and simplifications for the original plan. Several issues have been raised since then. This section details the current status for each one of the highlighted concerns.
### Separate bucket :white_check_mark:
We have identified a way to use a **single** bucket while preserving isolation between data and ensuring we can have online garbage collection from the start, which were the two main drivers behind the need for a separate bucket. This has been described in https://gitlab.com/gitlab-org/container-registry/-/issues/334#note_548292718.
### Separate instance/cluster :white_check_mark:
We have discussed and tested an alternative that requires a **single** registry instance/cluster and does not require managing separate processes/threads. This has been described in https://gitlab.com/gitlab-org/container-registry/-/issues/352.
### Separate migration service :white_check_mark:
We have proposed a different approach to triggering and tracking the migration of existing repositories without requiring a separate service or a custom client in https://gitlab.com/gitlab-org/container-registry/-/issues/364.
### Migration speed :white_check_mark:
This will be controlled by the number of workers for https://gitlab.com/gitlab-org/container-registry/-/issues/364.
epic