Dependency list page throws error if Container Scanning dependency report is present

Summary

A regression has been discovered which is caused by https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/3422+.

This results in the following behaviour:

If the compressed_package_metadata_query feature flag for the project is disabled:

The dependency list page throws an error and no entries show up:
If the compressed_package_metadata_query feature flag for the project is enabled:

Unknown licenses are displayed for certain dependencies in the gl-dependency-scanning-report.json file produced by container scanning.

Old behaviour: (before https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/3422+)

Licenses are displayed for both pytz dependencies:

New behaviour: (after https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/3422+)

Unknown license is displayed for pytz dependency from container scanning report:

Notes

Unfortunately, we've already shipped https://gitlab.com/gitlab-org/gitlab/-/issues/416654+, so we won't be able to fix this bug for the 16.2.2 backport.

The message Error fetching the dependency list. Please check your network connection and try again. is returned because the cte query produces the following invalid SQL due to the NULL purl_type values:

WITH needed_package_versions(purl_type, name, version) AS (VALUES (NULL, 'Django', '2.0'), (NULL, 'Django', '2.0'), (NULL, 'libssl1.1', '1.1.1u-r1'), (NULL, 'libcrypto1.1', '1.1.1u-r1'), (NULL, '.python-rundeps', '20230511.233342'), (NULL, 'alpine-baselayout', '3.4.3-r1'), (NULL, 'alpine-keys', '2.4-r1'), (NULL, 'apk-tools', '2.14.0-r2'), (NULL, 'binutils', '2.40-r7'), (NULL, 'brotli', '1.0.9-r14'), (NULL, 'busybox', '1.36.0-r9'), (NULL, 'ca-certificates', '20230506-r0'), (NULL, 'gcc', '12.2.1_git20220924-r10'), (NULL, 'gdbm', '1.23-r1'), (NULL, 'gmp', '6.2.1-r3'), (NULL, 'isl26', '0.26-r1'), (NULL, 'keyutils', '1.6.3-r3'), (NULL, 'krb5-conf', '1.0-r2'), (NULL, 'krb5', '1.20.1-r1'), (NULL, 'libaio', '0.3.113-r1')) SELECT DISTINCT "pm_packages"."id", "needed_package_versions"."version" FROM "needed_package_versions" INNER JOIN "pm_packages" ON "pm_packages"."purl_type" = "needed_package_versions"."purl_type" AND "pm_packages"."name" = "needed_package_versions"."name";

ERROR:  operator does not exist: smallint = text
LINE 1: ...R JOIN "pm_packages" ON "pm_packages"."purl_type" = "needed_...
                                                             ^
HINT:  No operator matches the given name and argument types. You might need to add explicit type casts.

Steps to reproduce

Create the following gl-dependency-scanning-report.json file:

{
  "version": "15.0.6",
  "scan": {
    "type": "dependency_scanning",
    "start_time": "2023-05-31T16:10:06",
    "end_time": "2023-05-31T16:10:08",
    "status": "success",
    "scanner": {
      "id": "trivy",
      "name": "Trivy",
      "url": "https://github.com/aquasecurity/trivy/",
      "vendor": {
        "name": "GitLab"
      },
      "version": "0.36.1"
    },
    "analyzer": {
      "id": "gcs",
      "name": "GitLab Container Scanning",
      "vendor": {
        "name": "GitLab"
      },
      "version": "6.0.1"
    }
  },
  "vulnerabilities": [],
  "dependency_files": [
    {
      "path": "some-image-path",
      "package_manager": "Python (python-pkg)",
      "dependencies": [
        {
          "package": {
            "name": "pytz"
          },
          "version": "2023.3"
        }
      ]
    }
  ]
}

Create the following gl-sbom-report.cdx.json file:

{
  "bomFormat": "CycloneDX",
  "specVersion": "1.4",
  "serialNumber": "urn:uuid:3e6effa5-5c5e-43d8-afa0-5b19b300cdf3",
  "version": 1,
  "metadata": {
    "timestamp": "2023-08-01T20:17:41+00:00",
    "tools": [
      {
        "vendor": "aquasecurity",
        "name": "trivy",
        "version": "0.36.1"
      }
    ]
  },
  "components": [
    {
      "bom-ref": "pkg:pypi/pytz@2023.3?file_path=usr%2Flocal%2Flib%2Fpython3.11%2Fsite-packages%2Fpytz-2023.3.dist-info%2FMETADATA",
      "type": "library",
      "name": "pytz",
      "version": "2023.3",
      "purl": "pkg:pypi/pytz@2023.3"
    }
  ],
  "dependencies": [],
  "vulnerabilities": []
}

Create the following .gitlab-ci.yml file:

my job:
  script:
    - echo test
  artifacts:
    reports:
      cyclonedx: "**/gl-sbom-*.cdx.json"
      dependency_scanning: gl-dependency-scanning-report.json

View the dependencies list for the project.

If the compressed_package_metadata_query feature flag is enabled for the project, notice that unknown license is displayed for pytz.

If the compressed_package_metadata_query feature flag is disabled for the project, the message Error fetching the dependency list. Please check your network connection and try again. is displayed.

Example Project

https://gitlab.com/adamcohen/dependency-list-pagination-container-scanning-regression/-/dependencies

What is the current bug behavior?

unknown license is displayed or error is shown on dependencies list page.

What is the expected correct behavior?

Actual licenses should be displayed on dependencies list page, for example: BSD 3-Clause "New" or "Revised" License, MIT License, unknown

Possible fixes

The reason for this error is due to the fact the rails backend is not properly converting the dependency_files[].package_manager field of the gl-dependency-scanning-report.json file produced by Container Scanning.

The rails backend converts from the container scanning package_manager to a purl_type using PACKAGE_MANAGER_TO_PURL_TYPE_MAP, however, this map is missing a few package managers that container scanning outputs:

analyzer (gobinary)
Python (python-pkg)
debian:12.1 (apt)
Java (jar)
alma:8.4 (unknown)
alpine:3.12.0 (apk)
amazon:2 (Karoo) (yum)
centos:8.4.2105 (yum)
debian:10.9 (apt)
debian:9.13 (apt)
cbl-mariner:2.0.20230630 (unknown)
opensuse.leap:15.0 (zypper)
oracle:8.2 (yum)
photon:1.0 (tdnf)
redhat:8.2 (yum)
redhat:7.9 (yum)
debian:9.4 (apt)
Java (jar)
alpine:3.18.2 (apk)
rocky:8.5 (unknown)
ubuntu:18.04 (apt)

We can fix this by updating the purl_type_for_pkg_manager method to convert package_manager values that can be mapped to purl_type values such as gobinary -> golang and python-pkg -> pypi, however, we'll need to figure out how to handle other values, such as yum, apk, zypper, unknown, etc.

In addition, we need to perform an audit of the different types of package managers that container scanning might produce in the dependency_files section, to make sure we haven't missed any.

Note: This was discussed here earlier.

/cc @atiwari71 @greg @gonzoyumo @ngeorge1

Edited Aug 07, 2023 by Adam Cohen