Add bulk partition-aware pipeline lookup and preloader
What does this MR do and why?
Stacked on top of !236791 (merged). ActiveRecord's standard preloader can already load a Ci::Pipeline association for a collection of records in a single WHERE id IN (...) query, but that query has no partition_id filter and therefore scans every partition of p_ci_pipelines. This is the unpruned cross-partition scan that triggered INC-8367 (see #593701 (closed)). This MR adds a custom preloader that uses the pipelines_id_range column on ci_partitions to add partition pruning to the bulk lookup.
Three small layers:
Ci::Partition.partition_ids_for(pipeline_ids)— single SQL roundtrip that resolves a list of pipeline ids to their tracked partition ids using thepipelines_id_rangecolumn. Returns[[pipeline_id, partition_id_or_nil], ...]. Future-proofed against an overrides table via aCOALESCEbranch.Gitlab::Ci::Pipeline::BulkByIdLookup— bulk analogue ofByIdLookup. Applies the same three-step cascade (current partition → range lookup → full scan) across an id array in at most three queries. Each fallback logs the count of unresolved ids.Ci::Preloaders::PipelinePreloader— AR-aware preloader that usesBulkByIdLookupand populates the association cache so subsequent reads issue zero queries. Acceptsassociation:andforeign_key:so the same class serveshead_pipeline,pipeline,ci_pipeline,latest_pipeline, etc.
The preloader composes with the per-record ByIdLookup override through ActiveRecord's loaded? short-circuit: list endpoints opt in to the preloader for batched loads, single-record reads still go through the per-record path.
No callers are converted yet — subsequent MRs will opt in list endpoints (REST, GraphQL, Sidekiq) one at a time.
References
- Related issue: #599815 (Add partition pruning to pipeline relations not in the CI database)
- Predecessor MR: !236791 (merged) (
Use pipelines_id_range to resolve Ci::Pipeline by id) - Origin issue: #593701 (closed)
Screenshots or screen recordings
N/A — no UI changes.
How to set up and validate locally
In rails console:
mrs = MergeRequest.where.not(head_pipeline_id: nil).limit(20).to_a
Ci::Preloaders::PipelinePreloader.new(mrs, association: :head_pipeline, foreign_key: :head_pipeline_id).preload_all
mrs.each(&:head_pipeline)Watch the console: preload_all issues at most three queries against p_ci_pipelines, each one filtered by partition_id, and the subsequent head_pipeline reads issue none.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.