Prune CI partitions for cross-pipeline dependencies
What does this MR do and why?
Cross-pipeline needs:pipeline:job lookups in Ci::BuildDependencies read every p_ci_pipelines and p_ci_builds partition.
Behind the ci_cross_pipeline_deps_partition_pruning feature flag, the lookup now runs in two steps:
Gitlab::Ci::PipelineSourceHierarchy#family_objectswalksci_sources_pipelinesonly. It returns(pipeline_id, partition_id)for the same-project ancestors and all their descendants.Ci::BuildDependencieskeeps the pairs that match the requested pipeline ids. A child pipeline walks up and then down. Any other pipeline walks down from itself.Ci::Buildis filtered by(commit_id, partition_id)for those pairs.
The same pairs answer the optional-dependency check in valid_cross_pipeline?.
With the flag off, the existing queries run.
References
- Resolves #631236
Database
Plans come from gitlab-production-ci, for root pipeline 2921930169 (partition 116) with six child pipelines. The job is rspec unit predictive in child pipeline 2921944020.
With the flag on, one BuildDependencies instance runs a pair query and a build query. With the flag off, it runs one build query.
Pair query: child pipeline
The seed is the child. The query looks up the root.
WITH RECURSIVE "ancestors" AS (
(SELECT 2921944020::bigint AS pipeline_id, 116::bigint AS partition_id FROM (SELECT 1) AS _seed)
UNION
(SELECT "ci_sources_pipelines"."source_pipeline_id" AS pipeline_id,
"ci_sources_pipelines"."source_partition_id" AS partition_id
FROM "ci_sources_pipelines", "ancestors"
WHERE "ci_sources_pipelines"."pipeline_id" = "ancestors"."pipeline_id"
AND "ci_sources_pipelines"."partition_id" = "ancestors"."partition_id"
AND "ci_sources_pipelines"."source_project_id" = "ci_sources_pipelines"."project_id")
), "descendants" AS (
(SELECT "pipeline_id", "partition_id" FROM "ancestors" AS "ci_sources_pipelines")
UNION
(SELECT "ci_sources_pipelines"."pipeline_id", "ci_sources_pipelines"."partition_id"
FROM "ci_sources_pipelines", "descendants"
WHERE "ci_sources_pipelines"."source_pipeline_id" = "descendants"."pipeline_id"
AND "ci_sources_pipelines"."source_partition_id" = "descendants"."partition_id"
AND "ci_sources_pipelines"."source_project_id" = "ci_sources_pipelines"."project_id")
)
SELECT "pipeline_id", "partition_id" FROM "descendants" AS "ci_sources_pipelines"
WHERE "ci_sources_pipelines"."pipeline_id" = 2921930169 LIMIT 1Plan: 2.0 ms, 5 buffer hits. It reads ci_sources_pipelines only.
Pair query: non-child pipeline
The seed is the root. The query looks up a child.
WITH RECURSIVE "descendants" AS (
(SELECT 2921930169::bigint AS pipeline_id, 116::bigint AS partition_id FROM (SELECT 1) AS _seed)
UNION
(SELECT "ci_sources_pipelines"."pipeline_id", "ci_sources_pipelines"."partition_id"
FROM "ci_sources_pipelines", "descendants"
WHERE "ci_sources_pipelines"."source_pipeline_id" = "descendants"."pipeline_id"
AND "ci_sources_pipelines"."source_partition_id" = "descendants"."partition_id"
AND "ci_sources_pipelines"."source_project_id" = "ci_sources_pipelines"."project_id")
)
SELECT "pipeline_id", "partition_id" FROM "descendants" AS "ci_sources_pipelines"
WHERE "ci_sources_pipelines"."pipeline_id" = 2921944020 LIMIT 1Plan: 14.7 ms, of which 12.6 ms is I/O on a cold read (6 reads).
Build query: flag on
SELECT "p_ci_builds".* FROM "p_ci_builds"
WHERE "p_ci_builds"."type" = 'Ci::Build'
AND ("p_ci_builds"."retried" = FALSE OR "p_ci_builds"."retried" IS NULL)
AND ("p_ci_builds"."commit_id" = 2921930169 AND "p_ci_builds"."partition_id" = 116
OR "p_ci_builds"."commit_id" = 2921944020 AND "p_ci_builds"."partition_id" = 116)
AND "p_ci_builds"."name" = 'rspec unit predictive'Plan: 6.0 ms, 11 buffer hits. It reads one partition, ci_builds_116.
Build query: flag off
This query embeds the pipeline hierarchy CTE and has no partition_id predicate. It reads 17 p_ci_builds partitions: planning 146.5 ms (median), execution about 1.4 ms, 211 buffer hits. SQL and plans: !253120 (merged)
Screenshots or screen recordings
Not applicable: no UI change.
How to set up and validate locally
- In a Rails console, find or create a same-project pipeline family with a trigger across two CI partitions. Add a build that depends on a job in the other partition.
- Run
Ci::BuildDependencies.new(build).allwithci_cross_pipeline_deps_partition_pruningenabled and disabled. Both runs return the requested job. - With the flag on, the build query contains
commit_idandpartition_idpredicates.
MR acceptance checklist
See the MR acceptance checklist.