Skip to content

Implement child pipeline architecture with resource groups for Container Registry PDM execution

Problem statement

As part of the Container Registry post deployment migrations (PDMs) automation effort (epic: gitlab-org&17933), we need to implement a robust pipeline architecture that prevents concurrent executions and ensures proper resource locking.

This issue is a follow-up to #21259 (closed).

Proposal

Implement a child pipeline architecture for executing PDMs in each environment (pre, staging, production) with the following characteristics:

Child Pipeline Structure

  • Parent Pipeline: Contains trigger jobs for each environment
  • Child Pipelines: One per environment (pre, staging, production) that actually executes the PDMs

Resource Group Implementation

  • Pipeline Lock: The trigger job in the parent pipeline should use a resource_group that serves as a "pipeline lock" to prevent multiple PDM pipelines from running simultaneously
  • Cluster Lock: The job in the child pipeline that actually applies changes to the cluster should use the cluster-level resource_group to prevent conflicts with other deployments

Benefits

The following benefits can be obtained once all pipelines in https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com use pipeline locks.

  • Prevents concurrent executions on environments
  • Enables automatic coordination with Release Managers and auto-deploy packages

Acceptance Criteria

  • Parent pipeline contains trigger jobs for each environment (pre, staging, production)
  • Each trigger job uses a resource_group for pipeline-level locking
  • Child pipeline jobs that apply cluster changes use cluster-level resource_group
  • Documentation is updated to explain the pipeline architecture
Edited by Reuben Pereira