Design and Test Container Registry PDM Automation in CI/CD Pipeline

Problem Description

Currently, post-deployment migrations (PDMs) for Container Registry on GitLab.com require manual intervention from SREs, creating a bottleneck for development. The Container Registry team is blocked on delivering changes because they can't deploy PDMs automatically. We need to design and test automation for this final piece of the deployment process to unblock the Container Registry team.

References:

• Current Container Registry deployment configuration
• Container Registry database migrations documentation

Solution Description

Previous solution description

We will design and test an extension to the existing CI/CD pipeline for Container Registry deployments to automatically trigger post-deployment migrations after all clusters have been successfully updated. This approach leverages the existing deployment infrastructure without requiring significant changes.

The solution will:

1. Add an additional job to the existing deployment pipeline that:

   • Runs after the regular deployment jobs have completed successfully
   • Sets SKIP_POST_DEPLOYMENT_MIGRATIONS = false (currently set to true)
   • Applies the Helm release again, causing only the migrations job to run with PDMs enabled
   • Includes appropriate error handling and reporting
   • Can be triggered manually via environment variable

2. Ensure this job runs only after the main stage deployment is complete

3. Enable manual pipeline creation through environment variables that control which jobs run in the pipeline

We will be creating a CI pipeline for executing Container Registry post deployment migrations progressively in pre, staging and production. Details can be seen in #21188 (comment 2669625253).

Exit Criteria

Previous exit criteria

• CI/CD pipeline extension designed and tested:
  • Job correctly sets SKIP_POST_DEPLOYMENT_MIGRATIONS = false
  • Job applies Helm release to trigger migrations
  • Job waits for migration completion and reports status
  • Manual triggering via environment variable implemented
• Pipeline creation via environment variables designed:
  • Environment variables control which jobs run in the pipeline
• Job sequencing designed to ensure PDMs run only after main stage deployment completes
• Design completed:
  • Successful test in preprod environment
  • Design validated and ready for staging/production deployment
• Documentation created:
  • Pipeline configuration documented
  • Runbooks created for knowledge and troubleshooting

• Design a solution to the problem of automating PDM execution for Container Registry
• Plan out the steps for achieving the solution