Ingest the dependency graph of third-party CycloneDX SBOMs

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Why are we doing this work

This issue adds the dependency graph to the components of Ingest components of third-party CycloneDX SBOMs (#633165), so that the Dependency List shows which packages of a third-party SBOM are direct and which are transitive, as it does for the GitLab DS analyzer's own SBOMs. This applies when a CI job uploads a CycloneDX JSON SBOM under artifacts:reports:cyclonedx.

In broad strokes, ingestion reads the dependencies array of the SBOM, which lists which package depends on which. It chooses a root for each manifest in the SBOM and marks a package direct when the root of its report depends on it.

This approach has three caveats. A whole-project SBOM holds the graph of every manifest in one place, so each report keeps only the edges of its own path, and each tool marks the root differently. An SBOM that records no dependency edges has no direct package, because direct is read from the edges. Rules that read only the file path and the graph leave out the rows of a module or of an aggregate manifest, such as a parent pom.xml that only lists modules, so a real package at such a file can be left out too.

Terms used in this issue:

  • A manifest is a file that declares packages, such as pom.xml
  • A report is the part of an SBOM for one path, such as one manifest, and each SBOM of the GitLab DS analyzer is one report
  • A row is one package at one path in the Dependency List
  • The root of a report is the node whose dependency edges start it, which for the GitLab DS analyzer is an anonymous node named Root Application
  • A top package is a package that no other package at its path depends on
  • A manifest node is a component with no package URL that stands for a manifest in the graph, such as the node Trivy writes for pom.xml
  • A module is a sub-project of a Maven or Gradle build with its own manifest, and an aggregate manifest is a manifest that only lists other manifests, such as the parent pom.xml of a Maven build
  • The default location is the fixed path 3rd Party Input File Path Unknown that Ingest components of third-party CycloneDX SBOMs (#633165) gives a package with no recorded source file

The diagrams below show how the graph root decides which packages are direct, today and with this issue. Yellow marks what this issue changes, and amber marks a gap today.

Graph today

Graph with this issue

Hard requirement

GitLab SBOM ingestion shows no changes. This means component data derived from GitLab SBOMs continues to look the same after changes in this issue are applied.

Requirements covered

The epic has no numbered requirement for the dependency graph. As a result, the requirement is listed here, from what the epic covers in its section "Ingest the dependency graph":

  • Each package is listed at the manifest that declares it
  • A package is direct when the root of its report depends on it
    • A package with no parent in its own report is also direct when the SBOM records dependency edges
  • The rows of a module and of an aggregate manifest are left out
  • An SBOM with no dependency edges has no direct package

What this issue adds for each format

A rule has no effect on an SBOM that lacks the field it reads.

Generic CycloneDX

  • A third-party SBOM covers a whole project in one graph, which starts from a node other than Root Application
    • So the root of each report is chosen from what the graph records, whatever the format, by the rules in "Choosing the root of each report"
    • A package is then direct or transitive by the rules in "Direct and transitive flags"

cdxgen

  • cdxgen writes a package once, with a source file property (internal:SrcFile in cdxgen 13, SrcFile in cdxgen 12) that holds each manifest that declares it, and lists its modules under metadata
    • So the root of a report is the node of its module where that node is not a component, and a package with no parent, such as core at foo/pom.xml, is direct
    • Each manifest in the property becomes one report, except an aggregate manifest such as the parent pom.xml, which gets no rows

Trivy

  • Trivy writes a manifest node for each manifest and lists a package once for each scope
    • So the root of each report is its manifest node, and the scope copies fold into one row
    • Each manifest node with the packages reachable from it becomes one report

Syft

  • Syft records each path of a package in its syft:location:N:path properties, and writes a manifest as a component of type file outside the dependency graph
    • So the manifest node rule does not apply, and the other root rules choose the root
    • Each path in the properties becomes one report, and a package with several paths is in the report of each

Discussion

Where the root rules run

The root of a report decides which packages read as direct, so a root rule must not change what the GitLab DS analyzer's own SBOMs show, or what an SBOM shows that lacks what the rule reads. The components issue, Ingest components of third-party CycloneDX SBOMs (#633165), adds a normalizing step, new code that runs during ingestion, after validation, and converts the components of known formats, with a rule for each format.

The root, direct and no-edge rules apply to every third-party SBOM, from what its graph records, and the normalizing step is kept only for format divergences in type and source file.

  • A rule for each format cannot express a root rule, because the choice depends on where a node sits in the graph and not on a property of the component
  • What the dependency graph records is a fact of the SBOM and not a claim about a format, so the rules need no recognized format and no step of their own beside the normalizing step

A package with no parent

In some SBOMs a package has no parent in its own report: nothing, including the root of its report, depends on it. The root choice cannot make such a package direct.

An example is core in the cdxgen sample cdxgen.cdx.json of this issue, a cdxgen SBOM of a Maven project with all 13 of its packages, which no package depends on.

Whether such a package counts as direct decides what the Dependency List shows for every module of a build.

A package with no parent is marked direct when the SBOM records dependency edges for other packages, and not direct when it records none.

  • The answer on an SBOM with no dependency edges keeps the GitLab DS analyzer's own marking, where a package is direct only when the root lists it
  • Where the SBOM records parents for other packages, a package with none is top level, so it is direct

A lockfile with one direct package and no manifest node

In an SBOM with no node for the manifest, a report with exactly one top package is ambiguous. It is either a file whose only direct package is that package, with a path such as poetry.lock, or a module whose own package has no root above it, with a path such as core/pom.xml. The path of the report tells them apart. A module's report is at a pom.xml or a Gradle build file (build.gradle, build.gradle.kts), the only files where a module is listed as a package. Every other path holds real packages. Choosing wrongly either hides a real direct package or shows a module as a package.

A lone top package of a report is kept as a direct row when the path of the report is not a pom.xml or a Gradle build file.

  • Leaving the package out removes a real direct package
  • A module's report is at a pom.xml or a Gradle build file, so its own package is still left out of the rows

Non-functional requirements

  • Documentation: the root rule, the direct flag rules, the SBOM with no edges and the rows that are left out are documented by Document the supported generators and the Cyclo... (#633171)
  • Feature flag: gated by the feature flag third_party_sbom_support of Ingest components of third-party CycloneDX SBOMs (#633165), disabled by default and read for the project of the job that uploads the SBOM
  • Performance: ingestion time of one large SBOM is recorded before and after
  • Testing: specs
    • A spec per rule
    • A spec per format on hand-trimmed SBOMs
    • A spec on an SBOM that lacks what each rule reads, expecting the same result as before
    • Each rule has a spec that fails with the rule disabled

Implementation plan

Splitting a whole-project SBOM into one report per manifest

Choosing the root of each report

  • Apply the root, direct and no-edge rules to every third-party SBOM, from what its graph records
  • Choose as the root of each report every node above a top package of the report that is not a package of the SBOM
    • This covers the manifest node, where the SBOM records one, and a module that the SBOM does not list as a component, such as metadata.component
    • Where the report has no such node, the only top package at a pom.xml or a Gradle build file is the root, so that the module's own package is left out of the rows
  • The packages a manifest declares can then read as direct
    • Trivy: the manifest node above the packages
    • cdxgen: the node of the module that lists the packages of the manifest, which cdxgen lists under metadata and not as a component, such as foo for foo/pom.xml in the cdxgen sample cdxgen.cdx.json of this issue
  • Never choose a node that is listed as a package at another path, so a module that another module declares keeps its row
  • Never choose the only top package of a report as its root when the path of the report is not a pom.xml or a Gradle build file (build.gradle, build.gradle.kts), so that package stays a direct row
    • Such paths are, for example, poetry.lock, package.json or packages.lock.json
  • Leave the root without a path where no rule applies, so that only the rule for a package with no parent applies

Direct and transitive flags

  • Mark a package direct when the root of its report depends on it, so the flags match the GitLab DS analyzer's own
  • Also mark a package direct when it has no parent in its own report and the SBOM records dependency edges
  • Leave a root out of the rows, so that a manifest or a module is not a dependency of itself

Packages with no parent and SBOMs with no edges

  • Apply the rules under "Direct and transitive flags" for a package with no parent in its own report and for an SBOM with no dependency edges
    • A package with no parent in its own report is direct when the SBOM records dependency edges
    • An SBOM with no dependency edges has no direct package

Dropping aggregate manifests and modules

  • Leave out the rows of an aggregate manifest
  • The row count then matches the GitLab DS analyzer's own
  • Leave out a module listed as a package at its own manifest, as the root of that manifest
  • The module is then not a row of itself

Limitations

  • The module rule tells a module from a dependency by the name of the file and the graph only
    • A report at a pom.xml or a Gradle build file whose one top package is a real dependency loses that row too
    • An example is commons-lang3, the only package at bar/pom.xml in cdxgen-12-srcfile-maven.cdx.json, a cdxgen 12.7.0 SBOM of a Maven project, which is taken for the module
    • That SBOM therefore gives 2 rows, where the GitLab DS analyzer gives 3 for the same project
  • The aggregate manifest rule reads only the directories of the paths, not the kind of file
    • A file whose every package is also a row or a module at a file in a deeper directory gets no rows, whatever its name
    • A root lockfile whose packages are all listed at files in deeper directories is therefore left out too
    • The default location is never an aggregate manifest

Acceptance criteria

General ingestion

  1. A package is direct when the root of its report depends on it, and transitive otherwise (example: commons-io direct and commons-lang3 transitive at core/pom.xml in trivy-reactor.cdx.json, a Trivy SBOM of a Maven project)

Exceptional cases

  1. An SBOM that is invalid against GitLab's CycloneDX schema is rejected as before, with the alert "Error parsing SBOM reports" (example: cdxgen-dotnet-empty-version.cdx.json, which has an empty version)
  2. A module is not a row of itself (example: no row for core at core/pom.xml in trivy-reactor.cdx.json)
  3. An aggregate manifest has no row (example: no row at pom.xml in trivy-reactor.cdx.json)
  4. A package with no parent is direct when the SBOM records dependency edges (example: core at foo/pom.xml in cdxgen.cdx.json attached to this issue, not the trimmed sample of the same name attached to Ingest components of third-party CycloneDX SBOMs (#633165))
  5. An SBOM that records no dependency edges has no direct package (example: syft.cdx.json)
  6. A package that the root of an SBOM lists is direct (example: syft-root-edge.cdx.json, the SBOM syft.cdx.json with an edge added from its root to one package)
  7. A package with several paths appears in the report of each path (example: syft-multi-path.cdx.json)
  8. A lone top package of a report at any path other than a pom.xml or a Gradle build file is a direct row (example: lockfile-single-top-package.cdx.json)
  9. A lone top package of a report at a pom.xml or a Gradle build file has no row (example: module-single-top-package.cdx.json)
  10. A report with no root candidate in an SBOM that records dependency edges has a package direct only when it has no parent (example: no-root-candidate.cdx.json)
  11. A package that Trivy lists once for each scope in which it is resolved gives one row at each path (example: log4j-core 2.14.1, which trivy-reactor.cdx.json lists four times, has one row at core/pom.xml and one at foo/pom.xml)

Unchanged behaviour

  1. With the feature flag disabled, a GitLab DS analyzer SBOM is ingested exactly as before the change, and with it enabled the SBOM keeps every row it shows before the change (example: gl-sbom-maven-core.cdx.json)
  2. With the feature flag enabled, the same SBOM keeps every direct flag it shows before the change (example: gl-sbom-maven-core.cdx.json)
  3. An SBOM from a generator with no special handling, with library components, gives its 3 rows (example: generic.cdx.json)

A per-sample checklist is kept with the attached samples.

Out of scope

Verification steps

These steps use the sample SBOMs attached to this issue. The same files are the fixtures of the merge requests, in ee/spec/fixtures/sbom/third_party/, where cdxgen.cdx.json is named cdxgen-reactor.cdx.json.

The cdxgen sample cdxgen.cdx.json of this issue holds all 13 packages that cdxgen 13.0.1 writes for a Maven project with the manifests pom.xml, core/pom.xml, foo/pom.xml and bar/pom.xml. The sample of the same name attached to Ingest components of third-party CycloneDX SBOMs (#633165) is trimmed to five packages and has no node for the module foo, so the criterion on core at foo/pom.xml does not hold with it.

  1. Check out the branch of the last merge request of this issue, on a GitLab Development Kit (GDK) instance with a plan that has dependency scanning, container scanning or license scanning (Ultimate)
  2. Create a test project and enable the feature flag for it
  3. Add a sample SBOM to the project and a .gitlab-ci.yml with a job that uploads it as artifacts:reports:cyclonedx
  4. Commit both on the default branch and wait for the pipeline to finish
  5. Open the Dependency List, read the location of each row, where a direct package shows (top level), and compare each row and its direct flag with the list of rows in the checklist of that sample
    • Any difference is a failure
  6. Repeat for each sample, and once for a GitLab DS analyzer SBOM such as gl-sbom-maven-core.cdx.json
    • The rows and direct flags of that SBOM must equal those from a project with the feature flag disabled
Edited by 🤖 GitLab Bot 🤖