Introduce image integration tests in Dependency Scanning analyzers
Summary
Introduce integration tests that check the Docker image and CLI of a Dependency Scanning analyzer project.
Compared to the existing job integration tests (AKA "QA jobs"), these new integration tests can be executed out of a CI job. As a consequence, they don't guarantee that the scanning job integrates properly with the CI. In particular, the Docker image might be incompatible in the job definition or with the CI itself.
Image integration tests are preferable to CLI integration tests because they better reproduce the running environment of the scanning job. That said, there might be cases where it's sufficient to test the analyzer CLI, and where executing docker run
only slows down the test with no benefit.
Improvements
It becomes easy to add tests that leverage the existing test projects:
- tests for supported environment variables (though it's not possible to check the default value set in the job definition)
- tests for when the supported files are moved to a sub-directory or sub-sub-directory
- tests using a custom vulnerability database, and resulting in a different report
Right now these tests can be implemented as job integration tests, but this involves creating branches in the Secure test projects, and maintaining them.
Also, it becomes possible to test scenarios where the analyzer fails with a specific exit code. At the moment this can't be achieved with the existing job integration tests. See #324963 (closed)
Risks
See Optional: Missing test coverage
Involved components
All Dependency Scanning analyzers.
Optional: Intended side effects
None
Optional: Missing test coverage
There's a possibly a gap between image and CLI integration tests and the corresponding job integration tests.
Possible gaps:
- The analyzer behaves differently because of the environment variables. This might be because of:
- predefined CI variables
- job variables, possibly defined in a job it extends
- variables set in the environment where the image or CLI integration tests are executed
- The scanning job is skipped but the analyzer CLI runs.
-
rules:exists
doesn't match any file of the scanned directory.- Filename doesn't match.
- Filename does match but directory depth doesn't.
-
Also, CLI and image integration tests need to be updated whenever the job script changes. If they're not updated, they no longer test the command that's being executed on the CI.
See Dependency Scanning CI template
Implementation plan
- Demonstrate image integration tests in the gemnasium project
-
List job integration tests that could be implemented as image integration tests -
Port these to image integration tests -
Optionally, port the other job integration tests, but skip them by default; see testing community contributions -
Update project dev documentation, and explain how to run image integration tests
-
-
Take notes on how to port job integration tests, in this issue - Communicate with Secure section
-
Share on Slack -
Share in a weekly meeting for the Secure stage
-