Implement child pipeline architecture with resource groups for Container Registry PDM execution
Problem statement
As part of the Container Registry post deployment migrations (PDMs) automation effort (epic: gitlab-org&17933), we need to implement a robust pipeline architecture that prevents concurrent executions and ensures proper resource locking.
This issue is a follow-up to #21259 (closed).
Proposal
Implement a child pipeline architecture for executing PDMs in each environment (pre, staging, production) with the following characteristics:
Child Pipeline Structure
- Parent Pipeline: Contains trigger jobs for each environment
- Child Pipelines: One per environment (pre, staging, production) that actually executes the PDMs
Resource Group Implementation
-
Pipeline Lock: The trigger job in the parent pipeline should use a
resource_groupthat serves as a "pipeline lock" to prevent multiple PDM pipelines from running simultaneously -
Cluster Lock: The job in the child pipeline that actually applies changes to the cluster should use the cluster-level
resource_groupto prevent conflicts with other deployments
Benefits
The following benefits can be obtained once all pipelines in https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com use pipeline locks.
- Prevents concurrent executions on environments
- Enables automatic coordination with Release Managers and auto-deploy packages
Acceptance Criteria
-
Parent pipeline contains trigger jobs for each environment (pre, staging, production) -
Each trigger job uses a resource_groupfor pipeline-level locking -
Child pipeline jobs that apply cluster changes use cluster-level resource_group -
Documentation is updated to explain the pipeline architecture
Edited by Reuben Pereira