Container Scanning QA doesn't check report structure
Summary
The end-to-end testing strategy does not detect schema changes to the clair scanner report. Upgrading the version of the clair dependencies requires manual verification that the schema of the report has not changed and will continue to work in the security dashboards.
V11 and V12 of the clair-scanner produced a report with the following structure:
{
"image": "ruby:2.6-alpine"
"unapproved": [],
"vulnerabilities": []
}
Example reports:
- https://gitlab-org.gitlab.io/-/security-products/tests/ruby-bundler/-/jobs/247935933/artifacts/gl-dependency-scanning-report.json
- https://gitlab-org.gitlab.io/-/security-products/tests/ruby-bundler/-/jobs/254558297/artifacts/gl-container-scanning-report.json
The ruby-bundler test project defines a report with the following schema:
{
"unapproved": [],
"vulnerabilities": []
}
The current testing strategy depends on taking the "unapproved" array from a generated report, sorting it and comparing it with the sorted "unapproved" array from the test report.
Steps to reproduce
- Trigger the pipeline associated with gitlab-org/security-products/tests/ruby-bundler!19 (closed)
Example Project
gitlab-org/security-products/tests/ruby-bundler!19 (closed)
What is the current bug behavior?
The build passes and ignores unexpected schema changes to the container scanning report. i.e. gl-container-scanning-report.json
What is the expected correct behavior?
The qa-container_scanning
job should detect schema changes and fail. Example
Relevant logs and/or screenshots
Downloading artifacts for dependency_scanning (264373441)...
Downloading artifacts from coordinator... ok id=264373441 responseStatus=200 OK token=shvShpQx
Downloading artifacts for container_scanning (264373442)...
Downloading artifacts from coordinator... ok id=264373442 responseStatus=200 OK token=W6r3xcQ9
Downloading artifacts for license_management (264373443)...
Downloading artifacts from coordinator... ok id=264373443 responseStatus=200 OK token=s5yAq7D6
$ apk update && apk add jq
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/community/x86_64/APKINDEX.tar.gz
v3.9.4-100-g98f4966a45 [http://dl-cdn.alpinelinux.org/alpine/v3.9/main]
v3.9.4-96-gd46d1b3369 [http://dl-cdn.alpinelinux.org/alpine/v3.9/community]
OK: 9774 distinct packages available
(1/2) Installing oniguruma (6.9.1-r0)
(2/2) Installing jq (1.6-r0)
Executing busybox-1.29.3-r10.trigger
OK: 7 MiB in 16 packages
$ jq -e --argfile a ./qa/expect/$REPORT --argfile b ./$REPORT -n '($a.unapproved | (.. | arrays) |= sort) as $a | ($b.unapproved | (.. | arrays) |= sort) as $b | $a == $b' || diff -b -u ./qa/expect/$REPORT ./$REPORT
true
Output of checks
This bug happens on GitLab.com