### Summary This proposal aims to improve CI pipeline execution latency by changing the urgency level of key pipeline execution workers from `:low` to `:high`. Currently, 15 core pipeline workers already have high urgency, but 7 workers directly involved in pipeline execution remain at low urgency, causing potential delays in the critical path of pipeline processing. As @hfyngvason observed in [INC-8246](https://app.incident.io/gitlab/incidents/8246) > I observed impact for pipeline creation and merge train processing, looking at merge train for gitlab-org/gitlab: > Head of the train is over 30 minutes old! Only 4 MRs have merge train pipelines so far. They should all have one (the limit is 20 concurrent pipelines) Things improved in the meantime but appear to be bad again. Merge train head is 11 minutes old now, and only the first entry even has a pipeline **Note**: We are working to address those saturation issues separately in https://gitlab.com/gitlab-com/feature-change-locks/-/work_items/86+ ### Background GitLab uses Sidekiq urgency levels to prioritize background job execution: - **`:high`** - Target: 10 seconds queue scheduling, 10 seconds execution - **`:low`** (default) - Target: 1 minute queue scheduling, 5 minutes execution - **`:throttled`** - No scheduling target, 5 minutes execution High urgency workers are intended for "latency-sensitive jobs that perform operations a user could reasonably expect to happen synchronously" (from `doc/development/sidekiq/worker_attributes.md`). Pipeline execution clearly falls into this category - users push code and expect pipelines to start and progress quickly. ### Current State #### Already High Urgency (15 workers) These workers in the `pipeline_processing` and `pipeline_creation` queues already have `:high` urgency: **Pipeline Processing:** - `PipelineProcessWorker` - Core pipeline state machine processing - `Ci::InitialPipelineProcessWorker` - Initial pipeline processing - `Ci::BuildFinishedWorker` - Build completion processing - `BuildQueueWorker` - Build queue updates - `Ci::CancelPipelineWorker` - Pipeline cancellation - `Ci::DropPipelineWorker` - Pipeline dropping - `Ci::RetryPipelineWorker` - Pipeline retry - `Ci::UserCancelPipelineWorker` - User-initiated cancellation - `Ci::PlayManualStageWorker` - Manual stage execution - `Ci::PipelineBridgeStatusWorker` - Bridge status updates - `UpdateHeadPipelineForMergeRequestWorker` - MR pipeline updates - `StageUpdateWorker` - Stage updates (deprecated) - `Ci::BuildTraceChunkFlushWorker` - Trace chunk flushing **Pipeline Creation:** - `CreatePipelineWorker` - Pipeline creation - `MergeRequests::CreatePipelineWorker` - MR pipeline creation - `Ci::ExternalPullRequests::CreatePipelineWorker` - External PR pipelines #### Currently Low Urgency (7 workers) These workers are currently at `:low` or default urgency despite being in the critical path: | Worker | Queue | Current Urgency | Purpose | File Location | |--------|-------|-----------------|---------|---------------| | `Ci::BuildPrepareWorker` | `pipeline_processing` | default (`:low`) | Prepares builds for execution | `app/workers/ci/build_prepare_worker.rb` | | `Ci::BuildScheduleWorker` | `pipeline_processing` | default (`:low`) | Executes scheduled builds | `app/workers/ci/build_schedule_worker.rb` | | `Ci::PipelineFinishedWorker` | `pipeline_processing` | `:low` | Post-pipeline completion tasks | `app/workers/ci/pipeline_finished_worker.rb` | | `Ci::ResourceGroups::AssignResourceFromResourceGroupWorker` | `pipeline_processing` | default (`:low`) | Assigns resources to waiting jobs | `app/workers/ci/resource_groups/assign_resource_from_resource_group_worker.rb` | | `RunPipelineScheduleWorker` | `pipeline_creation` | default (`:low`) | Creates scheduled pipelines | `app/workers/run_pipeline_schedule_worker.rb` | | `PipelineHooksWorker` | `pipeline_hooks` | default (`:low`) | Executes pipeline webhooks | `app/workers/pipeline_hooks_worker.rb` | | `Ci::PipelineArtifacts::CreateQualityReportWorker` | `pipeline_background` | `:low` | Creates quality reports | `app/workers/ci/pipeline_artifacts/create_quality_report_worker.rb` | ### Proposed Changes Change the following workers to `urgency :high`: #### Tier 1: Critical Execution Path (Recommended) These workers directly block pipeline/build execution and should be changed first: 1. **`Ci::BuildPrepareWorker`** - Blocks build execution until preparation completes - Users expect builds to start immediately after pipeline creation 2. **`Ci::BuildScheduleWorker`** - Controls when scheduled builds actually start - Users expect scheduled builds to start at the scheduled time 3. **`Ci::ResourceGroups::AssignResourceFromResourceGroupWorker`** - Jobs wait in pending state until resources are assigned - Delays here directly impact deployment execution time #### Tier 2: User-Facing Completion (Optional) These affect perceived completion time: 4. **`Ci::PipelineFinishedWorker`** - Users wait to see final pipeline status - Already has `data_consistency :sticky` and `idempotent!` - well-suited for high urgency 5. **`RunPipelineScheduleWorker`** - Scheduled pipelines (cron-triggered) should start promptly - Users expect scheduled pipelines to run at the scheduled time #### Excluded from High Urgency These should **NOT** be changed to high urgency: - **`PipelineHooksWorker`** ❌ - Reason: Calls external webhook URLs which may be slow or unavailable - High urgency workers cannot have external dependencies per `doc/development/sidekiq/worker_attributes.md:150` - Already in separate `pipeline_hooks` queue (weight 2) for isolation - **`Ci::PipelineArtifacts::CreateQualityReportWorker`** ❌ - Reason: Post-processing task, not critical path - Quality reports can be generated asynchronously - Already in `pipeline_background` queue (weight 1)