Use `Ci::Preloaders::PipelinePreloader` for bulk pipeline preloading on non-CI-DB models

Problem

!236832 (merged) introduced a partition-aware bulk preloader for Ci::Pipeline associations, deferring caller conversion: "No callers are converted yet — subsequent MRs will opt in list endpoints (REST, GraphQL, Sidekiq) one at a time."

ActiveRecord's standard preload / includes for belongs_to :pipeline associations on non-CI-DB models issues WHERE id IN (...) against p_ci_pipelines with no partition_id filter, causing a full cross-partition scan across every partition. As the number of CI partitions grows, this becomes progressively more expensive.

#599815 covers the single-record path via Ci::Partitionable::AssociationFinder#partitionable_belongs_to_loader. This issue covers the bulk path: making list endpoints (REST, GraphQL, Sidekiq) preload the pipeline association in a partition-aware way for the non-CI-DB models listed below.

Proposal

Note (approach changed): The standalone Ci::Preloaders::PipelinePreloader.new(...).preload_all API originally proposed here was removed in !240414 (merged). The bulk path is now a concern-level relation hook in Ci::Partitionable::AssociationFinder, gated behind the partition_aware_pipeline_preload feature flag (see #603166).

For each model, register the pipeline association in the loader and convert its list scope to chain with_partition_aware_preload:

include Ci::Partitionable::AssociationFinder

belongs_to :pipeline, class_name: 'Ci::Pipeline'
partitionable_belongs_to_loader :pipeline

scope :preload_pipeline, -> { with_partition_aware_preload.preload(:pipeline) }

with_partition_aware_preload extends the relation with PipelineRelationPreload, which overrides AR's preload_associations: it collects the registered foreign-key values, resolves them through Gitlab::Ci::Pipeline::BulkByIdLookup (partition-pruned), and feeds the results back via ActiveRecord::Associations::Preloader#available_records:. Nested child preloads (e.g. pipeline: { project: :namespace }) are preserved by the trailing super.

Behavior:

  • FF on (partition_aware_pipeline_preload): partition-pruned bulk pipeline lookup, no full cross-partition scan.
  • FF off: falls back to vanilla AR bulk preload (still not N+1), by design.

Scope review

After auditing all 18 models for an actual bulk pipeline-load path (a real collection/list endpoint that preloads the pipeline association), 8 models have no bulk path and need no change here. They already declare partitionable_belongs_to_loader (covered by #599815 for the single-record reader), but no list endpoint loads the pipeline association in bulk:

Model Why no bulk fix
Environments::Job pipeline association never read; only bulk inserts and reverse FK queries
Vulnerabilities::Statistic latest_pipeline_id is write-only; no endpoint exposes the pipeline
Vulnerabilities::PartialScan only EXISTS/FK-copy/pluck usage; association never read
Security::Scan single-record access only; GraphQL ScanType exposes no pipeline field
Sbom::OccurrenceRef pipeline_id write-only (ingestion); association never read
Dependencies::DependencyListExport single-record only (exportable); entity exposes no pipeline
Dast::ProfilesPipeline create-only join model; ci_pipeline never read in a list path
Dast::PreScanVerification single has_one on DastProfileType; type exposes no pipeline field

Progress

# Model Association FK column MR Status
— Framework (PipelineRelationPreload + FF) — — !240414 (merged) ✅ merged
1 MergeRequest::Metrics pipeline pipeline_id — ⬜ todo (per-record N+1 in MR REST entity)
2 Packages::BuildInfo pipeline pipeline_id — ⬜ todo (via Package#pipelines through-assoc)
3 Packages::PackageFileBuildInfo pipeline pipeline_id — ⬜ todo (via PackageFile#pipelines through-assoc)
4 Environments::Job pipeline ci_pipeline_id — ⏭️ no bulk path — skip
5 MergeTrains::Car pipeline pipeline_id !240901 (merged) ✅ merged
6 Security::PolicySchedulePipeline pipeline pipeline_id !240414 (merged) ✅ merged
7 Security::ScheduledPipelineExecutionPolicyTestRun pipeline pipeline_id !240433 🔄 in review
8 Vulnerabilities::Statistic pipeline latest_pipeline_id — ⏭️ no bulk path — skip
9 Vulnerabilities::Finding initial_finding_pipeline initial_pipeline_id — ⬜ todo (via VulnerabilitiesResolver lookahead)
10 Vulnerabilities::Finding latest_finding_pipeline latest_pipeline_id — ⬜ todo (via VulnerabilitiesResolver lookahead)
11 Vulnerabilities::PartialScan pipeline pipeline_id — ⏭️ no bulk path — skip
12 Vulnerabilities::Feedback pipeline pipeline_id !241111 (merged) ✅ merged
13 Security::Scan pipeline pipeline_id — ⏭️ no bulk path — skip
14 Sbom::Occurrence pipeline pipeline_id !241099 (merged) ✅ merged
15 Sbom::OccurrenceRef pipeline pipeline_id — ⏭️ no bulk path — skip
16 Dependencies::DependencyListExport pipeline pipeline_id — ⏭️ no bulk path — skip
17 Dast::ProfilesPipeline ci_pipeline ci_pipeline_id — ⏭️ no bulk path — skip
18 Dast::PreScanVerification ci_pipeline ci_pipeline_id — ⏭️ no bulk path — skip

Remaining actionable: MergeRequest::Metrics, Packages::BuildInfo, Packages::PackageFileBuildInfo, Vulnerabilities::Finding (both associations).

References

Edited by Marius Bobin