Testing graph exports to job artifacts when Gemnasium fails or when debugging
Related to #341215.
Availability & Testing
To be tested in 2-stage pipelines:
- The
test
stage runs the scanning job. The jobscript
is altered to make it fail with a specific exit code, and the job is allowed to fail on that exit code, so the pipeline isn't stopped. Theartifacts
param of the job isn't altered. - The
qa
stage runs a job that compares the CI artifacts/graph exports to what's expected. The Dependency Scanning report isn't checked.
The QA CI template can't be used for this: the artifacts
param shouldn't be altered because we wouldn't test the upload of the CI artifacts–this feature.
Scenarios to be covered:
- Graph exports are uploaded as CI artifacts when the
/analyzer
command returns a non-zero exit code.
Graph exports are uploaded as CI artifacts when the command returns a zero exit code and (Descoped in original ticket)SECURE_LOG_LEVEL
is debug
.
Graph exports are NOT uploaded when the command returns a zero exit code and SECURE_LOG_LEVEL
is info
(or not set).
These don't check the generated security report, so they can't be implemented using the existing Dependency Scanning QA CI template. Instead, we need a new pipeline with a CI job that checks the CI artifacts of the scanning job.
Testing
Technically we could add a spec for the Dependency Scanning (DS) CI template, in the gitlab
project. There we could check that paths of graph exports are declared as CI artifacts, in the definition of the DS jobs. However, we wouldn't check that graph exports created by the DS analyzers (running in Docker containers) are consistent with the artifacts
param of the corresponding DS jobs, and this is why I believe we need job integration tests. We need a failing scanning job that generates graph exports, and a check to ensure that the graph exports are exposed as CI artifacts.
This should be tested in complex/composite test projects that generate multiple graph exports:
- https://gitlab.com/gitlab-org/security-products/tests/java-gradle-multimodules and
- https://gitlab.com/gitlab-org/security-products/tests/java-maven-multimodules
- https://gitlab.com/gitlab-org/security-products/tests/scala-sbt-multiproject/
Tasks
-
Sync up on latest state of #341215 and assist if needed -
Job to test that given filename(s) are present, and if so, they are non-zero files -
Unit tests -
Put it all together - Test scenario (branches?) Graph exports are uploaded as CI artifacts when the /analyzer
command returns a non-zero exit code.