This epic is meant to store the project plans for the various stages of the container registry deployment and migration. ## Context GitLab is planning a significant upgrade to the container registry. This update will include support for online garbage collection, which will allow GitLab to reduce the cost of storage for the registry by up to 40%. In addition, by moving the storage of container image manifests to Postgres, we'll be able to measure and display how much storage a given project's registry is using, which will help customers better manage their storage costs, including GitLab.com ## Problem to solve We would like to have all projects/groups object storage usage for the container registry displayed on the usage quotas page. In order to achieve that, we will need to have completed the migration for all GitLab.com projects from using object storage for the image manifests to the new Postgres database. The problem is that it's a lot of data. At 5+ petabytes, migrating projects to the new registry will be a long process. We will encounter issues that need to be triaged and remediated. These issues will likely impact not just the Package group but the Support, Infrastructure, Delivery, and Datastores teams as well. This epic houses the various environment level project plans that have been requested. ## Success Metrics 1. 100% of GitLab.com projects migrated to the next generation Container Registry 1. Reduce total storage consumption for the Container Registry on GitLab.com by 40%. ## Rollout Strategy A gradual migration plan was proposed, discussed, and approved in https://gitlab.com/gitlab-org/container-registry/-/issues/374. We refer to the phases detailed in this plan whenever we mention Phase 1, 2, or 3 in this epic and sub-epics/issues. Please refer to it for additional details. ## DRIs This major effort requires involvement from multiple teams and departments across GitLab. Therefore, we have defined a list of primary and backup DRIs across those teams (see https://gitlab.com/gitlab-com/Product/-/issues/3431 - internal). These individuals are the point of contact for all related tasks assigned to their teams: | Group | Primary DRI | Backup DRI | Extra support | |-------|-------------|------------|---------------| | ~"group::package" | `@jdrpereira` | `@hswimelar` | `@jaime` | | ~"group::database" | `@stomlinson` | n/a [notes](https://gitlab.com/gitlab-com/Product/-/issues/3431#note_737233408) | | | ~"group::distribution" | `@WarheadsSE` | `@balasankarc` | `@pursultani` | |~"team::Delivery" | `@ggillies` | `@skarbek` | | | ~"team::Scalability" | `@qmnguyen0711` | `@reprazent` | | ## Timeline ### Phase 2 (Existing repositories) ```mermaid gantt dateFormat YYYY-MM-DD axisFormat %m-%d section Development excludes weekends section Staging P1 Development Work : done, dev1, 2022-01-25, 2022-03-01 P2 Development Work : done, dev2, after dev1, 10d P3 Development Work : done, dev3, after dev2, 18d Testing : done, stg1, 2022-03-10, 2022-04-09 section Production Readiness review : done, prd0, 2022-03-28, 2022-04-13 Deployment : done, prd1, after prd0, 1d section Rollout - GitLab Org Testing Testing : done, r1, after prd1, 2d section Rollout - free All Repositories : done, r2, after r1, 30d section Rollout - premium and ultimate All Repositories : done, r3, after r2, 42d section Rollout - VIP VIP Repositories : done, r4, after r3, 25d section Rollout - GitLab Org Remaining Remaining Repositories: r5, after r4, 9d section Rollout - Exceptional Repositories Exceptional Repositories : r6, after r3, 47d section Delete duplicate data from storage Quiet Period: r7, after r6, 14d section Delete duplicate data from storage Delete duplicate data: r8, after r7, 4d ``` <details><summary>Full detail</summary> Including past tasks: ```mermaid gantt dateFormat YYYY-MM-DD axisFormat %m-%d section Development excludes weekends section Staging P1 Development Work : done, dev1, 2022-01-25, 2022-03-01 P2 Development Work : done, dev2, after dev1, 10d P3 Development Work : done, dev3, after dev2, 18d Testing : done, stg1, 2022-03-10, 2022-04-09 section Production Readiness review : done, prd0, 2022-03-28, 2022-04-13 Deployment : done, prd1, after prd0, 1d section Rollout - GitLab Org Testing Testing : done, r1, after prd1, 2d section Rollout - free All Repositories done, : r2, after r1, 30d section Rollout - premium and ultimate All Repositories : done, r3, after r2, 30d section Rollout - VIP VIP Repositories done, : done, r4, after r3, 30d section Rollout - GitLab Org Remaining Remaining Repositories: r5, after r4, 4d section Delete duplicate data from storage Delete duplicate data: r6, after r5, 4d ``` </details> ## Updates - 2022-05-23 👉 https://gitlab.com/groups/gitlab-org/-/epics/5523#note_957122497