Determine whether to use Trivy or syft for generating SBOMs in Container Scanning

Proposal

In Generate SBOM in Container Scanning for Trivy-b... (#396381 - closed), we added support for generating SBOM files in Container Scanning for Trivy-based analyzers using trivy image --format cyclonedx <image>. We did a brief comparison of the different tools for generating SBOMs in this thread but decided to use trivy because it's already pre-installed for Trivy-based container scanning analyzers, and therefore was faster to implement than using another tool that needed to be downloaded separately.

Now that container scanning can produce SBOMs, we need to revisit the comparison of different SBOM generating tools so we can choose the best option.

Implementation Plan

  1. Compare trivy SBOM generation against syft in terms of the following (as well as any other criteria not listed that might be important):

    • Speed

      How do they compare in terms of speed? Is one faster than the other?

    • Data

      Is there extra data or features provided by one of the tools that isn't available in the other? For example, in this thread, it shows that syft provides CPE information, but how important is that for us?

    • Accuracy

      Is one more accurate than the other?

      As discussed in this thread, syft reports more components in the SBOM than trivy, although it's up for debate whether this makes it "more accurate". It seems that the extra components reported by syft are application packages as opposed to operating system packages.

      Use this code as a starting point to compare the results of trivy vs syft.

      You can find trivy and syft SBOM files for all of the distributions we support here.

      As an additional data point, it looks like trivy generates an sbom that contains cbl-mariner package type/protocol, for example pkg:cbl-mariner/bash@5.1.8-1.cm2?arch=x86_64, however, syft refers to this same component using rpm as the package type/protocol, for example: pkg:rpm/mariner/bash@5.1.8-1.cm2?arch=x86_64&upstream=bash-5.1.8-1.cm2.src.rpm&distro=mariner-2.0. We need to determine which one is a more accurate representation of this data.

  2. After making this comparison and determining which SBOM generating tool is better, we need to either replace the tool (if we choose syft) or if we stick with trivy, then no action needs to be taken.

Edited by Adam Cohen