Try to find pipeline/jobs in the current partition

Marius Bobinrequested to merge
593701-partition-pruning-ci-method into master

What does this MR do and why?

This MR introduces a Ci::PartitionableFinder concern that optimizes lookups for CI partitioned records (Ci::Pipeline and CommitStatus/jobs) by first attempting to find them in the current partition before falling back to a full cross-partition scan.

Problem

Several high-frequency Sidekiq workers query CI partitioned tables (p_ci_pipelines, p_ci_builds) without a partition_id, causing full cross-partition scans. As the number of CI partitions grows, these non-pruned queries become progressively more expensive. This was identified as a primary contributor to LockManager LWLock saturation on CI replicas (see incident INC-8367).

Solution

Introduces a new Ci::PartitionableFinder concern with a find_by_id class method that:

  1. Resolves the current partition ID via Ci::Partition.current
  2. First queries with partition_id scoped to the current partition (enabling partition pruning and avoiding cross-partition scans)
  3. Falls back to a full table scan only if the record is not found in the current partition (e.g. for older records in previous partitions)

This concern is included in both Ci::Pipeline and CommitStatus (the base class for all CI jobs/builds).

The behavior is gated behind the ci_partitionable_finder feature flag (gitlab_com_derisk type, disabled by default).

Performance impact

  • Happy path (record in current partition): 1 query with partition pruning instead of a full cross-partition scan
  • Fallback path (record in an older partition): 2 queries, same as before but with an optimistic first attempt

Changelog: fixed

References

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Marius Bobin

Merge request reports