Use Ci::Preloaders::PipelinePreloader for bulk pipeline preloading on non-CI-DB models

Problem

!236832 (merged) introduced a partition-aware bulk preloader for Ci::Pipeline associations, deferring caller conversion: "No callers are converted yet — subsequent MRs will opt in list endpoints (REST, GraphQL, Sidekiq) one at a time."

ActiveRecord's standard preload / includes for belongs_to :pipeline associations on non-CI-DB models issues WHERE id IN (...) against p_ci_pipelines with no partition_id filter, causing a full cross-partition scan across every partition. As the number of CI partitions grows, this becomes progressively more expensive.

#599815 (closed) covers the single-record path via Ci::Partitionable::AssociationFinder#partitionable_belongs_to_loader. This issue covers the bulk path: making list endpoints (REST, GraphQL, Sidekiq) preload the pipeline association in a partition-aware way for the non-CI-DB models listed below.

Proposal

Note (approach changed): The standalone Ci::Preloaders::PipelinePreloader.new(...).preload_all API originally proposed here was removed in !240414 (merged). The bulk path is now a concern-level relation hook in Ci::Partitionable::AssociationFinder, gated behind the partition_aware_pipeline_preload feature flag (see #603166).

For each model, register the pipeline association in the loader and convert its list scope to chain with_partition_aware_preload:

include Ci::Partitionable::AssociationFinder

belongs_to :pipeline, class_name: 'Ci::Pipeline'
partitionable_belongs_to_loader :pipeline

scope :preload_pipeline, -> { with_partition_aware_preload.preload(:pipeline) }

with_partition_aware_preload extends the relation with PipelineRelationPreload, which overrides AR's preload_associations: it collects the registered foreign-key values, resolves them through Gitlab::Ci::Pipeline::BulkByIdLookup (partition-pruned), and feeds the results back via ActiveRecord::Associations::Preloader#available_records:. Nested child preloads (e.g. pipeline: { project: :namespace }) are preserved by the trailing super.

Behavior:

  • FF on (partition_aware_pipeline_preload): partition-pruned bulk pipeline lookup, no full cross-partition scan.
  • FF off: falls back to vanilla AR bulk preload (still not N+1), by design.

Scope review

After auditing all 18 models for an actual bulk pipeline-load path (a real collection/list endpoint that preloads the pipeline association), 8 models have no bulk path and need no change here. They already declare partitionable_belongs_to_loader (covered by #599815 (closed) for the single-record reader), but no list endpoint loads the pipeline association in bulk:

Model Why no bulk fix
Environments::Job pipeline association never read; only bulk inserts and reverse FK queries
Vulnerabilities::Statistic latest_pipeline_id is write-only; no endpoint exposes the pipeline
Vulnerabilities::PartialScan only EXISTS/FK-copy/pluck usage; association never read
Security::Scan single-record access only; GraphQL ScanType exposes no pipeline field
Sbom::OccurrenceRef pipeline_id write-only (ingestion); association never read
Dependencies::DependencyListExport single-record only (exportable); entity exposes no pipeline
Dast::ProfilesPipeline create-only join model; ci_pipeline never read in a list path
Dast::PreScanVerification single has_one on DastProfileType; type exposes no pipeline field

Deferred: nested / through-association models

The four models originally marked actionable all reach the pipeline through an intermediate association, not a direct belongs_to on the loaded relation. PipelineRelationPreload#preload_associations only prunes when the extended relation's own class registered the loader (in partitioned_pipeline_loaders), so chaining with_partition_aware_preload on the top relation is a no-op for these — the pipeline lives one level down:

Model Bulk path Blocker
MergeRequest::Metrics MergeRequest → metrics → pipeline (REST entity) nested preload
Packages::BuildInfo Package → build_infos → pipeline (has_many :pipelines, through:) through-association
Packages::PackageFileBuildInfo PackageFile → package_file_build_infos → pipeline (through) through-association
Vulnerabilities::Finding Vulnerability → findings → {initial,latest}_finding_pipeline (resolver lookahead) nested preload

No correctness gap: per-record pipeline reads on all four are already partition-pruned via the single-record reader override from #599815 (closed). Only the bulk preload optimization is missing, and it would require extending the framework to prune nested/through pipeline keys. Deferring until a specific list endpoint proves hot enough to justify that framework change; will track in a dedicated follow-up if/when needed.

Progress

# Model Association FK column MR Status
— Framework (PipelineRelationPreload + FF) — — !240414 (merged) ✅ merged
1 MergeRequest::Metrics pipeline pipeline_id — ⏸️ deferred — nested preload (MR→metrics→pipeline), needs framework support
2 Packages::BuildInfo pipeline pipeline_id — ⏸️ deferred — through-assoc, needs framework support
3 Packages::PackageFileBuildInfo pipeline pipeline_id — ⏸️ deferred — through-assoc, needs framework support
4 Environments::Job pipeline ci_pipeline_id — ⏭️ no bulk path — skip
5 MergeTrains::Car pipeline pipeline_id !240901 (merged) ✅ merged
6 Security::PolicySchedulePipeline pipeline pipeline_id !240414 (merged) ✅ merged
7 Security::ScheduledPipelineExecutionPolicyTestRun pipeline pipeline_id !240433 (merged) 🔄 in review
8 Vulnerabilities::Statistic pipeline latest_pipeline_id — ⏭️ no bulk path — skip
9 Vulnerabilities::Finding initial_finding_pipeline initial_pipeline_id — ⏸️ deferred — nested preload, needs framework support
10 Vulnerabilities::Finding latest_finding_pipeline latest_pipeline_id — ⏸️ deferred — nested preload, needs framework support
11 Vulnerabilities::PartialScan pipeline pipeline_id — ⏭️ no bulk path — skip
12 Vulnerabilities::Feedback pipeline pipeline_id !241111 (merged) ✅ merged
13 Security::Scan pipeline pipeline_id — ⏭️ no bulk path — skip
14 Sbom::Occurrence pipeline pipeline_id !241099 (merged) ✅ merged
15 Sbom::OccurrenceRef pipeline pipeline_id — ⏭️ no bulk path — skip
16 Dependencies::DependencyListExport pipeline pipeline_id — ⏭️ no bulk path — skip
17 Dast::ProfilesPipeline ci_pipeline ci_pipeline_id — ⏭️ no bulk path — skip
18 Dast::PreScanVerification ci_pipeline ci_pipeline_id — ⏭️ no bulk path — skip

Remaining actionable: none — the four models above are deferred (all reach pipeline via a nested/through association the current framework cannot prune; per-record reads are already pruned via #599815 (closed)). Will revisit if a bulk path proves hot enough to justify extending the framework.

References

Edited by Marius Bobin