Batch Execution for Scheduled Pipeline Execution Policies

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

Problem Statement

When dealing with scheduled pipeline executions across a large number of projects, the current random time window distribution may not provide enough control over resource utilization. Users need more precise control over how and when pipelines are triggered across their projects.

Business Value

• Better control over CI/CD resource consumption
• Improved predictability of pipeline execution timing
• Reduced risk of resource exhaustion
• Enhanced ability to manage large-scale pipeline executions
• Optimized scheduling for different organizational needs

User Story

As a DevOps engineer or security team member,
I want to run scheduled pipelines in controlled batches,
So that I can better manage resources and predict execution patterns when dealing with a large number of projects.

Requirements

Core Functionality

1. New batch execution configuration options:

   • Number of projects to process in each batch
   • Time delay between batch executions
   • Option to switch between batch execution and random time window distribution

2. Batch Processing:

   • Trigger pipelines for the first batch at scheduled time
   • Process subsequent batches after configured delay
   • Continue until all projects are processed
   • Calculate and show estimated completion time

3. UI Enhancements:

   • Add batch configuration options to policy editor
   • Display batch execution schedule
   • Show progress of batch processing
   • Indicate estimated completion time

4. Resource Management:

   • Validate batch size inputs
   • Ensure efficient processing of large project sets
   • Prevent resource exhaustion

Authorization

• Maintain existing permission model for policy management
• Follow GitLab's standard authorization patterns

Out of Scope

• Dynamic batch size adjustment
• Batch prioritization
• Integration with other scheduling features
• Custom batch ordering

Rollout

• Feature flag protected
• Gradual rollout to ensure stability
• Monitor system impact and resource usage