Discuss steps for enabling auto-deploy for container registry

Summary

We are in a phase of frequently rolling out substantial container registry code changes for the container registry metadata migration. We do not run auto-deployments for registry yet, as we still are collecting experience with running container registry with a metadata DB and want to carefully test changes in staging first before deploying to production.

As of now, rolling out new versions of the registry requires many manual steps:

• CNG version bump (requiring review by distribution team)
• k8s-workloads version bump for each environment (requiring review and monitoring of deployment by infra team, taking care to not clash with ongoing auto-deploys)

This issue is to discuss ways to come to automated deployments of the registry DB to reduce toil while still providing high reliability.

Goals

• define roadmap to gain confidence in automated tests for registry deployments
• discuss possible solutions for deployment automation improvements
• discuss what's missing to add registry to our regular auto-deploy pipeline

High-level proposal

Task	Type	Depends on	Description	Target	Team(s)
A	Automation		New CI job in gitlab-org/container-registry to programatically create version bump MR in gitlab-org/build/CNG for every new git tag. The MR description should include the relevant changelog excerpt (delta between current version in CNG and tag version).	GitLab.com; Self-managed	Package
B	Test; Automation	A	- Extend tests in gitlab-org/build/CNG to cover the database automation functionality; - Automatically trigger test suite for version bump MRs.	GitLab.com; Self-managed	Distribution Package (?)
C	Automation	B	Once B proves to be stable, automatically set MWPS for gitlab-org/build/CNG version bump MRs.	GitLab.com; Self-managed	Distribution; Package
D	Automation	C	Repeat A, B and C but for gitlab-org/omnibus-gitlab and gitlab-org/charts version bumps?	Self-managed	Distribution; Package
E	Automation		Implement methodology for post-deployment database migrations	GitLab.com	Distribution; Package
F	Development; Test		Setup environment in Cloud Sandbox (HackyStack). This will allow the development team to share a "realistic" environment in GCP for advanced testing that can't be covered with integration and/or QA tests, and for F.	GitLab.com; Self-managed	Package
G	Test	F	Run all integration tests (not the QA/end-to-end tests) as part of the gitlab-org/container-registry CI pipeline. Currently we're not running the GCS and CDN tests that are part of the registry codebase because they require a GCP infrastructure to run against. We can rely on the setup from F for this (open to better solutions).	GitLab.com; Self-managed	Package
H	Test		Expand QA test coverage for the Container Registry (gitlab-org&5082 (closed))	GitLab.com; Self-managed	Package; Quality
I	Development; Operation		- Design and implement feature flags solution. Ideally, this should be applicable to all Go applications at GitLab (using the registry as PoC); - Integrate feature flags solution with chatops.	GitLab.com	Package; Delivery; Others
J	Automation	C I	Automatically create new tag in gitlab-org/container-registry for the latest green pipeline on master.	GitLab.com	Delivery
L	Automation	J	Adapt CI job from A so that it responds to events from I only.	GitLab.com; Self-managed	Package
M	Automation	L	Adjust wait job that polls for new images to be available, so that it can monitor and detect new registry images in gitlab-org/build/CNG.	GitLab.com	Delivery?
N	Automation	M	Adjust gitlab-com/gl-infra/k8s-workloads/gitlab-com to pick the latest auto deploy image tag for the registry and kick off the deployment.	GitLab.com	Delivery