Prune CI partitions for cross-pipeline dependencies

What does this MR do and why?

Cross-pipeline needs:pipeline:job lookups in Ci::BuildDependencies read every p_ci_pipelines and p_ci_builds partition.

Behind the ci_cross_pipeline_deps_partition_pruning feature flag, the lookup now runs in two steps:

  1. Gitlab::Ci::PipelineSourceHierarchy#family_objects walks ci_sources_pipelines only. It returns (pipeline_id, partition_id) for the same-project ancestors and all their descendants. Ci::BuildDependencies keeps the pairs that match the requested pipeline ids. A child pipeline walks up and then down. Any other pipeline walks down from itself.
  2. Ci::Build is filtered by (commit_id, partition_id) for those pairs.

The same pairs answer the optional-dependency check in valid_cross_pipeline?.

With the flag off, the existing queries run.

References

Database

Plans come from gitlab-production-ci, for root pipeline 2921930169 (partition 116) with six child pipelines. The job is rspec unit predictive in child pipeline 2921944020.

With the flag on, one BuildDependencies instance runs a pair query and a build query. With the flag off, it runs one build query.

Pair query: child pipeline

The seed is the child. The query looks up the root.

WITH RECURSIVE "ancestors" AS (
  (SELECT 2921944020::bigint AS pipeline_id, 116::bigint AS partition_id FROM (SELECT 1) AS _seed)
  UNION
  (SELECT "ci_sources_pipelines"."source_pipeline_id" AS pipeline_id,
          "ci_sources_pipelines"."source_partition_id" AS partition_id
   FROM "ci_sources_pipelines", "ancestors"
   WHERE "ci_sources_pipelines"."pipeline_id" = "ancestors"."pipeline_id"
     AND "ci_sources_pipelines"."partition_id" = "ancestors"."partition_id"
     AND "ci_sources_pipelines"."source_project_id" = "ci_sources_pipelines"."project_id")
), "descendants" AS (
  (SELECT "pipeline_id", "partition_id" FROM "ancestors" AS "ci_sources_pipelines")
  UNION
  (SELECT "ci_sources_pipelines"."pipeline_id", "ci_sources_pipelines"."partition_id"
   FROM "ci_sources_pipelines", "descendants"
   WHERE "ci_sources_pipelines"."source_pipeline_id" = "descendants"."pipeline_id"
     AND "ci_sources_pipelines"."source_partition_id" = "descendants"."partition_id"
     AND "ci_sources_pipelines"."source_project_id" = "ci_sources_pipelines"."project_id")
)
SELECT "pipeline_id", "partition_id" FROM "descendants" AS "ci_sources_pipelines"
WHERE "ci_sources_pipelines"."pipeline_id" = 2921930169 LIMIT 1

Plan: 2.0 ms, 5 buffer hits. It reads ci_sources_pipelines only.

Pair query: non-child pipeline

The seed is the root. The query looks up a child.

WITH RECURSIVE "descendants" AS (
  (SELECT 2921930169::bigint AS pipeline_id, 116::bigint AS partition_id FROM (SELECT 1) AS _seed)
  UNION
  (SELECT "ci_sources_pipelines"."pipeline_id", "ci_sources_pipelines"."partition_id"
   FROM "ci_sources_pipelines", "descendants"
   WHERE "ci_sources_pipelines"."source_pipeline_id" = "descendants"."pipeline_id"
     AND "ci_sources_pipelines"."source_partition_id" = "descendants"."partition_id"
     AND "ci_sources_pipelines"."source_project_id" = "ci_sources_pipelines"."project_id")
)
SELECT "pipeline_id", "partition_id" FROM "descendants" AS "ci_sources_pipelines"
WHERE "ci_sources_pipelines"."pipeline_id" = 2921944020 LIMIT 1

Plan: 14.7 ms, of which 12.6 ms is I/O on a cold read (6 reads).

Build query: flag on

SELECT "p_ci_builds".* FROM "p_ci_builds"
WHERE "p_ci_builds"."type" = 'Ci::Build'
  AND ("p_ci_builds"."retried" = FALSE OR "p_ci_builds"."retried" IS NULL)
  AND ("p_ci_builds"."commit_id" = 2921930169 AND "p_ci_builds"."partition_id" = 116
    OR "p_ci_builds"."commit_id" = 2921944020 AND "p_ci_builds"."partition_id" = 116)
  AND "p_ci_builds"."name" = 'rspec unit predictive'

Plan: 6.0 ms, 11 buffer hits. It reads one partition, ci_builds_116.

Build query: flag off

This query embeds the pipeline hierarchy CTE and has no partition_id predicate. It reads 17 p_ci_builds partitions: planning 146.5 ms (median), execution about 1.4 ms, 211 buffer hits. SQL and plans: !253120 (merged)

Screenshots or screen recordings

Not applicable: no UI change.

How to set up and validate locally

  1. In a Rails console, find or create a same-project pipeline family with a trigger across two CI partitions. Add a build that depends on a job in the other partition.
  2. Run Ci::BuildDependencies.new(build).all with ci_cross_pipeline_deps_partition_pruning enabled and disabled. Both runs return the requested job.
  3. With the flag on, the build query contains commit_id and partition_id predicates.

MR acceptance checklist

See the MR acceptance checklist.

Edited by Marius Bobin

Merge request reports

Loading
Loading