## Problem !236832 introduced a partition-aware bulk preloader for `Ci::Pipeline` associations, deferring caller conversion: *"No callers are converted yet — subsequent MRs will opt in list endpoints (REST, GraphQL, Sidekiq) one at a time."* ActiveRecord's standard `preload` / `includes` for `belongs_to :pipeline` associations on non-CI-DB models issues `WHERE id IN (...)` against `p_ci_pipelines` with no `partition_id` filter, causing a full cross-partition scan across every partition. As the number of CI partitions grows, this becomes progressively more expensive. #599815 covers the **single-record** path via `Ci::Partitionable::AssociationFinder#partitionable_belongs_to_loader`. This issue covers the **bulk** path: making list endpoints (REST, GraphQL, Sidekiq) preload the `pipeline` association in a partition-aware way for the non-CI-DB models listed below. ## Proposal > **Note (approach changed):** The standalone `Ci::Preloaders::PipelinePreloader.new(...).preload_all` API originally proposed here was **removed** in !240414. The bulk path is now a concern-level relation hook in `Ci::Partitionable::AssociationFinder`, gated behind the `partition_aware_pipeline_preload` feature flag (see #603166). For each model, register the pipeline association in the loader and convert its list scope to chain `with_partition_aware_preload`: ```ruby include Ci::Partitionable::AssociationFinder belongs_to :pipeline, class_name: 'Ci::Pipeline' partitionable_belongs_to_loader :pipeline scope :preload_pipeline, -> { with_partition_aware_preload.preload(:pipeline) } ``` `with_partition_aware_preload` extends the relation with `PipelineRelationPreload`, which overrides AR's `preload_associations`: it collects the registered foreign-key values, resolves them through `Gitlab::Ci::Pipeline::BulkByIdLookup` (partition-pruned), and feeds the results back via `ActiveRecord::Associations::Preloader#available_records:`. Nested child preloads (e.g. `pipeline: { project: :namespace }`) are preserved by the trailing `super`. Behavior: - **FF on** (`partition_aware_pipeline_preload`): partition-pruned bulk pipeline lookup, no full cross-partition scan. - **FF off**: falls back to vanilla AR bulk preload (still not N+1), by design. ## Scope review After auditing all 18 models for an actual bulk pipeline-load path (a real collection/list endpoint that preloads the pipeline association), **8 models have no bulk path and need no change here**. They already declare `partitionable_belongs_to_loader` (covered by #599815 for the single-record reader), but no list endpoint loads the pipeline association in bulk: | Model | Why no bulk fix | |---|---| | `Environments::Job` | `pipeline` association never read; only bulk inserts and reverse FK queries | | `Vulnerabilities::Statistic` | `latest_pipeline_id` is write-only; no endpoint exposes the pipeline | | `Vulnerabilities::PartialScan` | only `EXISTS`/FK-copy/pluck usage; association never read | | `Security::Scan` | single-record access only; GraphQL `ScanType` exposes no pipeline field | | `Sbom::OccurrenceRef` | `pipeline_id` write-only (ingestion); association never read | | `Dependencies::DependencyListExport` | single-record only (`exportable`); entity exposes no pipeline | | `Dast::ProfilesPipeline` | create-only join model; `ci_pipeline` never read in a list path | | `Dast::PreScanVerification` | single `has_one` on `DastProfileType`; type exposes no pipeline field | ## Progress | # | Model | Association | FK column | MR | Status | |---|---|---|---|---|---| | — | Framework (`PipelineRelationPreload` + FF) | — | — | !240414 | ✅ merged | | 1 | `MergeRequest::Metrics` | `pipeline` | `pipeline_id` | — | ⬜ todo (per-record N+1 in MR REST entity) | | 2 | `Packages::BuildInfo` | `pipeline` | `pipeline_id` | — | ⬜ todo (via `Package#pipelines` through-assoc) | | 3 | `Packages::PackageFileBuildInfo` | `pipeline` | `pipeline_id` | — | ⬜ todo (via `PackageFile#pipelines` through-assoc) | | 4 | `Environments::Job` | `pipeline` | `ci_pipeline_id` | — | ⏭️ no bulk path — skip | | 5 | `MergeTrains::Car` | `pipeline` | `pipeline_id` | !240901 | ✅ merged | | 6 | `Security::PolicySchedulePipeline` | `pipeline` | `pipeline_id` | !240414 | ✅ merged | | 7 | `Security::ScheduledPipelineExecutionPolicyTestRun` | `pipeline` | `pipeline_id` | !240433 | 🔄 in review | | 8 | `Vulnerabilities::Statistic` | `pipeline` | `latest_pipeline_id` | — | ⏭️ no bulk path — skip | | 9 | `Vulnerabilities::Finding` | `initial_finding_pipeline` | `initial_pipeline_id` | — | ⬜ todo (via `VulnerabilitiesResolver` lookahead) | | 10 | `Vulnerabilities::Finding` | `latest_finding_pipeline` | `latest_pipeline_id` | — | ⬜ todo (via `VulnerabilitiesResolver` lookahead) | | 11 | `Vulnerabilities::PartialScan` | `pipeline` | `pipeline_id` | — | ⏭️ no bulk path — skip | | 12 | `Vulnerabilities::Feedback` | `pipeline` | `pipeline_id` | !241111 | ✅ merged | | 13 | `Security::Scan` | `pipeline` | `pipeline_id` | — | ⏭️ no bulk path — skip | | 14 | `Sbom::Occurrence` | `pipeline` | `pipeline_id` | !241099 | ✅ merged | | 15 | `Sbom::OccurrenceRef` | `pipeline` | `pipeline_id` | — | ⏭️ no bulk path — skip | | 16 | `Dependencies::DependencyListExport` | `pipeline` | `pipeline_id` | — | ⏭️ no bulk path — skip | | 17 | `Dast::ProfilesPipeline` | `ci_pipeline` | `ci_pipeline_id` | — | ⏭️ no bulk path — skip | | 18 | `Dast::PreScanVerification` | `ci_pipeline` | `ci_pipeline_id` | — | ⏭️ no bulk path — skip | Remaining actionable: `MergeRequest::Metrics`, `Packages::BuildInfo`, `Packages::PackageFileBuildInfo`, `Vulnerabilities::Finding` (both associations). ## References - Bulk preloader framework: !236832, !240414 - Single-record pruning: #599815 - FF rollout: #603166 - Origin issue: #593701