Include child pipeline SBOMs in pipeline Licenses tab

What does this MR do and why?

The pipeline Licenses tab now reads SBOM (software bill of materials) reports from same-project child pipelines, behind the feature flag include_child_pipeline_sboms_in_pipeline_licenses.

SbomScanner only read SBOM reports from the pipeline's own builds, while the pipeline Security tab includes child pipelines. When the CycloneDX report was generated in a child pipeline, the parent pipeline's Licenses tab was hidden, and its licenses and license count JSON endpoints returned 404 (the HTML endpoint redirected to the pipeline).

Child pipelines are an opt-in keyword that defaults to false. The Licenses tab opts in, and the merge request license widget keeps reading only the pipeline's own builds. The license_count cache key now includes the sorted ids of the hierarchy's latest SBOM builds, so a child pipeline that adds an SBOM later or a flag toggle does not serve a stale count, and plain pipeline updates don't create new cache entries.

Flag type is gitlab_com_derisk, default off.

Also included: license approval policy evaluation

This MR now also includes !260896 (merged) (license approval policy evaluation reads child pipeline SBOMs, behind the license_policy_child_pipeline_sboms feature flag). @dbalexandre reviewed it and merged it into this stack base. The Licenses tab part above is unchanged.

References

Screenshots or screen recordings

Not captured; backend-only change to tab visibility/data — see validation output.

How to set up and validate locally

  1. Add a parent .gitlab-ci.yml with a trigger: include: child.yml job. In child.yml, add a job that uploads artifacts:reports:cyclonedx with one pypi component.
  2. In rails console, enable the flag:
    Feature.enable(:include_child_pipeline_sboms_in_pipeline_licenses)
  3. Open the parent pipeline. The Licenses tab appears with the license.

Local GDK result for a parent pipeline whose CycloneDX report is only in a child pipeline:

flag off: tab has_data?=false licenses=0 | default(policy/MR) has_data?=false licenses=0
flag on : tab has_data?=true licenses=1 | default(policy/MR) has_data?=false licenses=0

Specs: sbom_scanner_spec, pipeline_components_spec, license_scanning_spec, pipeline_helper_spec, EE pipelines_controller_spec and license_compliance_spec: 164 examples, 0 failures. ee/spec/models/ee/project_spec.rb #license_compliance: 7 examples, 0 failures. update_license_approvals_service_spec and branch_components_spec were green on a prior run (path unchanged).

Database

license_count builds its cache key with Ci::Pipeline#sbom_report_build_ids_in_self_and_project_descendants when include_child_pipeline_sboms_in_pipeline_licenses is on. The flag-off key is unchanged. Before f1cd9f95 the flag-on key already ran a hierarchy query on every request (self_and_project_descendants.maximum(:updated_at)), and scanner.has_data? runs the same hierarchy load plus a builds/artifacts EXISTS check just before it in both versions.

Plans from gitlab-production-ci on a merge request pipeline hierarchy of 5 same-project pipelines (parent + 4 children) with 320 latest builds and 1 SBOM build. Warm = run again after the cold run.

Query Used by Cold exec Warm exec Warm buffers
Hierarchy CTE on ci_sources_pipelines both keys, has_data? 19.0 ms 0.11 ms 25 hits
MAX(updated_at) over hierarchy pipelines old key only 18.6 ms 0.14 ms 25 hits
Load hierarchy pipelines + source_pipeline/source_bridge preloads new key, has_data? 1.2 / 24.7 / 23.9 ms 0.14 / 0.07 / 0.06 ms 25 / 16 / 8 hits
Latest SBOM build ids (below) new key only 453 ms (283 reads) 1.0 ms 1,555 hits
has_data? EXISTS (LIMIT 1) both keys 0.44 ms 441 hits

The new build-ids query is an index scan on ci_builds_<partition>_commit_id_type_ref_idx plus one index-only probe per latest build on ci_job_artifacts_<partition>_job_id_file_type_partition_id_idx, so its cost grows with the number of latest builds in the hierarchy. The cold time is I/O on the builds pages; in a request has_data? reads the same builds index just before.

SQL and plan (latest SBOM build ids)
SELECT "p_ci_builds"."id" FROM "p_ci_builds"
WHERE "p_ci_builds"."type" = 'Ci::Build'
  AND ("p_ci_builds"."retried" = FALSE OR "p_ci_builds"."retried" IS NULL)
  AND "p_ci_builds"."commit_id" IN (<hierarchy_pipeline_ids>)
  AND "p_ci_builds"."partition_id" = <partition_id>
  AND (EXISTS (SELECT 1 FROM "p_ci_job_artifacts"
    WHERE "p_ci_job_artifacts"."job_id" = "p_ci_builds"."id"
      AND "p_ci_job_artifacts"."partition_id" = "p_ci_builds"."partition_id"
      AND "p_ci_job_artifacts"."file_type" = 28))
Nested Loop  (cost=1.15..1997.95 rows=29 width=8) (actual time=0.313..0.978 rows=1 loops=1)
  Buffers: shared hit=1555
  ->  Index Scan using ci_builds_<partition>_commit_id_type_ref_idx on ci_builds_<partition> p_ci_builds  (actual rows=320 loops=1)
        Index Cond: ((commit_id = ANY (<hierarchy_pipeline_ids>)) AND ((type)::text = 'Ci::Build'::text))
        Filter: (((NOT retried) OR (retried IS NULL)) AND (partition_id = <partition_id>))
        Rows Removed by Filter: 1
        Buffers: shared hit=274
  ->  Index Only Scan using ci_job_artifacts_<partition>_job_id_file_type_partition_id_idx on ci_job_artifacts_<partition> p_ci_job_artifacts  (actual rows=0 loops=320)
        Index Cond: ((job_id = p_ci_builds.id) AND (file_type = 28) AND (partition_id = <partition_id>))
        Heap Fetches: 0
        Buffers: shared hit=1281
SQL (old key, for comparison)
SELECT MAX("p_ci_pipelines"."updated_at") FROM "p_ci_pipelines"
WHERE ("p_ci_pipelines"."id" = <pipeline_id> AND "p_ci_pipelines"."partition_id" = <partition_id>) OR ...

Bitmap OR of 5 primary-key lookups on ci_pipelines_<partition>.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Reviewable LOC: 74 (plus 171 in tests)

Context for LLM agents
  • Rejected: changing the shared scanner default so child pipelines are always included. SbomScanner and PipelineComponents also feed MR license approval policy evaluation (UpdateLicenseApprovalsService) and the MR license widget. Policy evaluation opts in explicitly via the stacked !260896 (merged) behind license_policy_child_pipeline_sboms.
  • Invariant: callers other than the Licenses tab helper, PipelinesController#licenses, #license_count, Project#license_compliance when called from licenses, and the approval policy evaluation from !260896 (merged) must keep self_and_project_descendants: false.
  • Invariant: the license_count cache key includes the hierarchy's sorted SBOM build ids only when the option is on, so flag-off keys stay identical to master.
  • Scope is same-project child pipelines (self_and_project_descendants), not multi-project downstream pipelines.

Owed elsewhere

  • License approval policy evaluation reading only the parent pipeline's SBOMs on the target branch (pre-existing licenses counted as newly detected): #632877, fixed by the included !260896 (merged). The MR license compliance widget has the same blind spot (MergeRequest#report_type_enabled? checks only the head pipeline's own SBOMs) and is not changed here; no issue tracks it yet.
Edited by Alan (Maciej) Paruszewski

Merge request reports

Loading
Loading