Replace job tests with image tests in Dependency Scanning analyzer projects
Summary
Update the Dependency Scanning analyzer projects to replace job integration tests (using external test projects and running in separate downstream pipelines) with image integration tests (using rspec and all running in the same image test
job).
Proposal
For each downstream pipeline using a branch of a Secure test project:
- check whether a job integration tests is strictly needed
- if not, copy the files from the test project to the
qa/fixtures
directory of the analyzer project; the sub-directory name istest_project_name/test_case_name
, and the default test case is "main" - add an RSpec group that replicates what's tested in the downstream pipeline
- make sure that the new RSpec examples are executed when running the
image test
job - remove the downstream pipeline from the CI config
Keep at least one job integration test, to check the job definition.
Image integration tests are implemented using the integration-test project. See gemnasium's spec/image_spec.rb.
Implementation plan
Repeat the steps of the proposal for all Dependency Scanning analyzer projects:
- https://gitlab.com/gitlab-org/security-products/analyzers/bundler-audit
- https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium
- https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven
- https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-python
- https://gitlab.com/gitlab-org/security-products/analyzers/retire.js
Improvements
- Image integration tests can run locally.
- They are easier to maintained, compared to job integration tests using external test projects.
- They run faster because there's no need to spin off a new job for each test case.
Risks
We might miss a case where the job definition can't be ignored. For instance, the image test might not set all the environment variables set in the scanning job. See #330093 (closed).
It might take longer to execute the pipeline if too many long-running tests are combined in a single image test
job. This might be a problem in the case of gemnasium-maven, gemnasium-python and retire.js because they build the project.
There might be leftover when building a project during the test, with possible side-effects. That's because the DockerRunner
of the integration-test reuses the fixtures directory. TODO: create an issue about cleaning up the fixtures directory after running an RSpec group, or to make the runner copy the fixture to a temporary directory every time it runs.
Involved components
- https://gitlab.com/gitlab-org/security-products/analyzers/bundler-audit
- https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium
- https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven
- https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-python
- https://gitlab.com/gitlab-org/security-products/analyzers/retire.js
Optional: Intended side effects
See improvements
Optional: Missing test coverage
See risks
Testing
The integration tests provide coverage. SET may need to refactor, replace or remove the Test Orchestrator.