## Problem !236832 introduced a partition-aware bulk preloader for `Ci::Pipeline` associations, deferring caller conversion: *"No callers are converted yet — subsequent MRs will opt in list endpoints (REST, GraphQL, Sidekiq) one at a time."* ActiveRecord's standard `preload` / `includes` for `belongs_to :pipeline` associations on non-CI-DB models issues `WHERE id IN (...)` against `p_ci_pipelines` with no `partition_id` filter, causing a full cross-partition scan across every partition. As the number of CI partitions grows, this becomes progressively more expensive. #599815 covers the **single-record** path via `Ci::Partitionable::AssociationFinder#partitionable_belongs_to_loader`. This issue covers the **bulk** path: making list endpoints (REST, GraphQL, Sidekiq) preload the `pipeline` association in a partition-aware way for the non-CI-DB models listed below. ## Proposal > **Note (approach changed):** The standalone `Ci::Preloaders::PipelinePreloader.new(...).preload_all` API originally proposed here was **removed** in !240414. The bulk path is now a concern-level relation hook in `Ci::Partitionable::AssociationFinder`, gated behind the `partition_aware_pipeline_preload` feature flag (see #603166). For each model, register the pipeline association in the loader and convert its list scope to chain `with_partition_aware_preload`: ```ruby include Ci::Partitionable::AssociationFinder belongs_to :pipeline, class_name: 'Ci::Pipeline' partitionable_belongs_to_loader :pipeline scope :preload_pipeline, -> { with_partition_aware_preload.preload(:pipeline) } ``` `with_partition_aware_preload` extends the relation with `PipelineRelationPreload`, which overrides AR's `preload_associations`: it collects the registered foreign-key values, resolves them through `Gitlab::Ci::Pipeline::BulkByIdLookup` (partition-pruned), and feeds the results back via `ActiveRecord::Associations::Preloader#available_records:`. Nested child preloads (e.g. `pipeline: { project: :namespace }`) are preserved by the trailing `super`. Behavior: - **FF on** (`partition_aware_pipeline_preload`): partition-pruned bulk pipeline lookup, no full cross-partition scan. - **FF off**: falls back to vanilla AR bulk preload (still not N+1), by design. ## Scope review After auditing all 18 models for an actual bulk pipeline-load path (a real collection/list endpoint that preloads the pipeline association), **8 models have no bulk path and need no change here**. They already declare `partitionable_belongs_to_loader` (covered by #599815 for the single-record reader), but no list endpoint loads the pipeline association in bulk: | Model | Why no bulk fix | |---|---| | `Environments::Job` | `pipeline` association never read; only bulk inserts and reverse FK queries | | `Vulnerabilities::Statistic` | `latest_pipeline_id` is write-only; no endpoint exposes the pipeline | | `Vulnerabilities::PartialScan` | only `EXISTS`/FK-copy/pluck usage; association never read | | `Security::Scan` | single-record access only; GraphQL `ScanType` exposes no pipeline field | | `Sbom::OccurrenceRef` | `pipeline_id` write-only (ingestion); association never read | | `Dependencies::DependencyListExport` | single-record only (`exportable`); entity exposes no pipeline | | `Dast::ProfilesPipeline` | create-only join model; `ci_pipeline` never read in a list path | | `Dast::PreScanVerification` | single `has_one` on `DastProfileType`; type exposes no pipeline field | ## Deferred: nested / through-association models The four models originally marked actionable all reach the pipeline through an **intermediate association**, not a direct `belongs_to` on the loaded relation. `PipelineRelationPreload#preload_associations` only prunes when the **extended relation's own class** registered the loader (in `partitioned_pipeline_loaders`), so chaining `with_partition_aware_preload` on the top relation is a no-op for these — the pipeline lives one level down: | Model | Bulk path | Blocker | |---|---|---| | `MergeRequest::Metrics` | `MergeRequest → metrics → pipeline` (REST entity) | nested preload | | `Packages::BuildInfo` | `Package → build_infos → pipeline` (`has_many :pipelines, through:`) | through-association | | `Packages::PackageFileBuildInfo` | `PackageFile → package_file_build_infos → pipeline` (through) | through-association | | `Vulnerabilities::Finding` | `Vulnerability → findings → {initial,latest}_finding_pipeline` (resolver lookahead) | nested preload | **No correctness gap:** per-record pipeline reads on all four are already partition-pruned via the single-record reader override from #599815. Only the bulk *preload* optimization is missing, and it would require extending the framework to prune nested/through pipeline keys. Deferring until a specific list endpoint proves hot enough to justify that framework change; will track in a dedicated follow-up if/when needed. ## Progress | # | Model | Association | FK column | MR | Status | |---|---|---|---|---|---| | — | Framework (`PipelineRelationPreload` + FF) | — | — | !240414 | ✅ merged | | 1 | `MergeRequest::Metrics` | `pipeline` | `pipeline_id` | — | ⏸️ deferred — nested preload (MR→metrics→pipeline), needs framework support | | 2 | `Packages::BuildInfo` | `pipeline` | `pipeline_id` | — | ⏸️ deferred — through-assoc, needs framework support | | 3 | `Packages::PackageFileBuildInfo` | `pipeline` | `pipeline_id` | — | ⏸️ deferred — through-assoc, needs framework support | | 4 | `Environments::Job` | `pipeline` | `ci_pipeline_id` | — | ⏭️ no bulk path — skip | | 5 | `MergeTrains::Car` | `pipeline` | `pipeline_id` | !240901 | ✅ merged | | 6 | `Security::PolicySchedulePipeline` | `pipeline` | `pipeline_id` | !240414 | ✅ merged | | 7 | `Security::ScheduledPipelineExecutionPolicyTestRun` | `pipeline` | `pipeline_id` | !240433 | 🔄 in review | | 8 | `Vulnerabilities::Statistic` | `pipeline` | `latest_pipeline_id` | — | ⏭️ no bulk path — skip | | 9 | `Vulnerabilities::Finding` | `initial_finding_pipeline` | `initial_pipeline_id` | — | ⏸️ deferred — nested preload, needs framework support | | 10 | `Vulnerabilities::Finding` | `latest_finding_pipeline` | `latest_pipeline_id` | — | ⏸️ deferred — nested preload, needs framework support | | 11 | `Vulnerabilities::PartialScan` | `pipeline` | `pipeline_id` | — | ⏭️ no bulk path — skip | | 12 | `Vulnerabilities::Feedback` | `pipeline` | `pipeline_id` | !241111 | ✅ merged | | 13 | `Security::Scan` | `pipeline` | `pipeline_id` | — | ⏭️ no bulk path — skip | | 14 | `Sbom::Occurrence` | `pipeline` | `pipeline_id` | !241099 | ✅ merged | | 15 | `Sbom::OccurrenceRef` | `pipeline` | `pipeline_id` | — | ⏭️ no bulk path — skip | | 16 | `Dependencies::DependencyListExport` | `pipeline` | `pipeline_id` | — | ⏭️ no bulk path — skip | | 17 | `Dast::ProfilesPipeline` | `ci_pipeline` | `ci_pipeline_id` | — | ⏭️ no bulk path — skip | | 18 | `Dast::PreScanVerification` | `ci_pipeline` | `ci_pipeline_id` | — | ⏭️ no bulk path — skip | Remaining actionable: none — the four models above are deferred (all reach pipeline via a nested/through association the current framework cannot prune; per-record reads are already pruned via #599815). Will revisit if a bulk path proves hot enough to justify extending the framework. ## References - Bulk preloader framework: !236832, !240414 - Single-record pruning: #599815 - FF rollout: #603166 - Origin issue: #593701