Dependency list page throws error if Container Scanning dependency report is present
Summary
A regression has been discovered which is caused by https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/3422+.
This results in the following behaviour:
-
If the
compressed_package_metadata_query
feature flag for the project is disabled:The dependency list page throws an error and no entries show up:
-
If the
compressed_package_metadata_query
feature flag for the project is enabled:Unknown licenses are displayed for certain dependencies in the
gl-dependency-scanning-report.json
file produced by container scanning.Old behaviour: (before https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/3422+)
Licenses are displayed for both
pytz
dependencies:New behaviour: (after https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/3422+)
Unknown license is displayed for
pytz
dependency from container scanning report:
Notes
Unfortunately, we've already shipped https://gitlab.com/gitlab-org/gitlab/-/issues/416654+, so we won't be able to fix this bug for the 16.2.2
backport.
The message Error fetching the dependency list. Please check your network connection and try again.
is returned because the cte query produces the following invalid SQL due to the NULL
purl_type
values:
WITH needed_package_versions(purl_type, name, version) AS (VALUES (NULL, 'Django', '2.0'), (NULL, 'Django', '2.0'), (NULL, 'libssl1.1', '1.1.1u-r1'), (NULL, 'libcrypto1.1', '1.1.1u-r1'), (NULL, '.python-rundeps', '20230511.233342'), (NULL, 'alpine-baselayout', '3.4.3-r1'), (NULL, 'alpine-keys', '2.4-r1'), (NULL, 'apk-tools', '2.14.0-r2'), (NULL, 'binutils', '2.40-r7'), (NULL, 'brotli', '1.0.9-r14'), (NULL, 'busybox', '1.36.0-r9'), (NULL, 'ca-certificates', '20230506-r0'), (NULL, 'gcc', '12.2.1_git20220924-r10'), (NULL, 'gdbm', '1.23-r1'), (NULL, 'gmp', '6.2.1-r3'), (NULL, 'isl26', '0.26-r1'), (NULL, 'keyutils', '1.6.3-r3'), (NULL, 'krb5-conf', '1.0-r2'), (NULL, 'krb5', '1.20.1-r1'), (NULL, 'libaio', '0.3.113-r1')) SELECT DISTINCT "pm_packages"."id", "needed_package_versions"."version" FROM "needed_package_versions" INNER JOIN "pm_packages" ON "pm_packages"."purl_type" = "needed_package_versions"."purl_type" AND "pm_packages"."name" = "needed_package_versions"."name";
ERROR: operator does not exist: smallint = text
LINE 1: ...R JOIN "pm_packages" ON "pm_packages"."purl_type" = "needed_...
^
HINT: No operator matches the given name and argument types. You might need to add explicit type casts.
Steps to reproduce
-
Create the following
gl-dependency-scanning-report.json
file:{ "version": "15.0.6", "scan": { "type": "dependency_scanning", "start_time": "2023-05-31T16:10:06", "end_time": "2023-05-31T16:10:08", "status": "success", "scanner": { "id": "trivy", "name": "Trivy", "url": "https://github.com/aquasecurity/trivy/", "vendor": { "name": "GitLab" }, "version": "0.36.1" }, "analyzer": { "id": "gcs", "name": "GitLab Container Scanning", "vendor": { "name": "GitLab" }, "version": "6.0.1" } }, "vulnerabilities": [], "dependency_files": [ { "path": "some-image-path", "package_manager": "Python (python-pkg)", "dependencies": [ { "package": { "name": "pytz" }, "version": "2023.3" } ] } ] }
-
Create the following
gl-sbom-report.cdx.json
file:{ "bomFormat": "CycloneDX", "specVersion": "1.4", "serialNumber": "urn:uuid:3e6effa5-5c5e-43d8-afa0-5b19b300cdf3", "version": 1, "metadata": { "timestamp": "2023-08-01T20:17:41+00:00", "tools": [ { "vendor": "aquasecurity", "name": "trivy", "version": "0.36.1" } ] }, "components": [ { "bom-ref": "pkg:pypi/pytz@2023.3?file_path=usr%2Flocal%2Flib%2Fpython3.11%2Fsite-packages%2Fpytz-2023.3.dist-info%2FMETADATA", "type": "library", "name": "pytz", "version": "2023.3", "purl": "pkg:pypi/pytz@2023.3" } ], "dependencies": [], "vulnerabilities": [] }
-
Create the following
.gitlab-ci.yml
file:my job: script: - echo test artifacts: reports: cyclonedx: "**/gl-sbom-*.cdx.json" dependency_scanning: gl-dependency-scanning-report.json
-
View the dependencies list for the project.
If the
compressed_package_metadata_query
feature flag is enabled for the project, notice thatunknown
license is displayed forpytz
.If the
compressed_package_metadata_query
feature flag is disabled for the project, the messageError fetching the dependency list. Please check your network connection and try again.
is displayed.
Example Project
https://gitlab.com/adamcohen/dependency-list-pagination-container-scanning-regression/-/dependencies
What is the current bug behavior?
unknown
license is displayed or error is shown on dependencies list page.
What is the expected correct behavior?
Actual licenses should be displayed on dependencies list page, for example: BSD 3-Clause "New" or "Revised" License
, MIT License
, unknown
Possible fixes
The reason for this error is due to the fact the rails backend is not properly converting the dependency_files[].package_manager
field of the gl-dependency-scanning-report.json
file produced by Container Scanning.
The rails backend converts from the container scanning package_manager
to a purl_type
using PACKAGE_MANAGER_TO_PURL_TYPE_MAP, however, this map is missing a few package managers that container scanning outputs:
- analyzer (gobinary)
- Python (python-pkg)
- debian:12.1 (apt)
- Java (jar)
- alma:8.4 (unknown)
- alpine:3.12.0 (apk)
- amazon:2 (Karoo) (yum)
- centos:8.4.2105 (yum)
- debian:10.9 (apt)
- debian:9.13 (apt)
- cbl-mariner:2.0.20230630 (unknown)
- opensuse.leap:15.0 (zypper)
- oracle:8.2 (yum)
- photon:1.0 (tdnf)
- redhat:8.2 (yum)
- redhat:7.9 (yum)
- debian:9.4 (apt)
- Java (jar)
- alpine:3.18.2 (apk)
- rocky:8.5 (unknown)
- ubuntu:18.04 (apt)
We can fix this by updating the purl_type_for_pkg_manager method to convert package_manager
values that can be mapped to purl_type
values such as gobinary -> golang
and python-pkg -> pypi
, however, we'll need to figure out how to handle other values, such as yum
, apk
, zypper
, unknown
, etc.
In addition, we need to perform an audit of the different types of package managers that container scanning might produce in the dependency_files
section, to make sure we haven't missed any.
Note: This was discussed here earlier.