Add bulk partition-aware pipeline lookup and preloader

What does this MR do and why?

Stacked on top of !236791 (merged). ActiveRecord's standard preloader can already load a Ci::Pipeline association for a collection of records in a single WHERE id IN (...) query, but that query has no partition_id filter and therefore scans every partition of p_ci_pipelines. This is the unpruned cross-partition scan that triggered INC-8367 (see #593701 (closed)). This MR adds a custom preloader that uses the pipelines_id_range column on ci_partitions to add partition pruning to the bulk lookup.

Three small layers:

  • Ci::Partition.partition_ids_for(pipeline_ids) — single SQL roundtrip that resolves a list of pipeline ids to their tracked partition ids using the pipelines_id_range column. Returns [[pipeline_id, partition_id_or_nil], ...]. Future-proofed against an overrides table via a COALESCE branch.
  • Gitlab::Ci::Pipeline::BulkByIdLookup — bulk analogue of ByIdLookup. Applies the same three-step cascade (current partition → range lookup → full scan) across an id array in at most three queries. Each fallback logs the count of unresolved ids.
  • Ci::Preloaders::PipelinePreloader — AR-aware preloader that uses BulkByIdLookup and populates the association cache so subsequent reads issue zero queries. Accepts association: and foreign_key: so the same class serves head_pipeline, pipeline, ci_pipeline, latest_pipeline, etc.

The preloader composes with the per-record ByIdLookup override through ActiveRecord's loaded? short-circuit: list endpoints opt in to the preloader for batched loads, single-record reads still go through the per-record path.

No callers are converted yet — subsequent MRs will opt in list endpoints (REST, GraphQL, Sidekiq) one at a time.

References

  • Related issue: #599815 (Add partition pruning to pipeline relations not in the CI database)
  • Predecessor MR: !236791 (merged) (Use pipelines_id_range to resolve Ci::Pipeline by id)
  • Origin issue: #593701 (closed)

Screenshots or screen recordings

N/A — no UI changes.

How to set up and validate locally

In rails console:

mrs = MergeRequest.where.not(head_pipeline_id: nil).limit(20).to_a

Ci::Preloaders::PipelinePreloader.new(mrs, association: :head_pipeline, foreign_key: :head_pipeline_id).preload_all

mrs.each(&:head_pipeline)

Watch the console: preload_all issues at most three queries against p_ci_pipelines, each one filtered by partition_id, and the subsequent head_pipeline reads issue none.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Marius Bobin

Merge request reports

Loading