Skip to content

Replace job tests with image tests in Dependency Scanning analyzer projects

Summary

Update the Dependency Scanning analyzer projects to replace job integration tests (using external test projects and running in separate downstream pipelines) with image integration tests (using rspec and all running in the same image test job).

Proposal

For each downstream pipeline using a branch of a Secure test project:

  • check whether a job integration tests is strictly needed
  • if not, copy the files from the test project to the qa/fixtures directory of the analyzer project; the sub-directory name is test_project_name/test_case_name, and the default test case is "main"
  • add an RSpec group that replicates what's tested in the downstream pipeline
  • make sure that the new RSpec examples are executed when running the image test job
  • remove the downstream pipeline from the CI config

Keep at least one job integration test, to check the job definition.

Image integration tests are implemented using the integration-test project. See gemnasium's spec/image_spec.rb.

Implementation plan

Repeat the steps of the proposal for all Dependency Scanning analyzer projects:

Improvements

  • Image integration tests can run locally.
  • They are easier to maintained, compared to job integration tests using external test projects.
  • They run faster because there's no need to spin off a new job for each test case.

Risks

We might miss a case where the job definition can't be ignored. For instance, the image test might not set all the environment variables set in the scanning job. See #330093 (closed).

It might take longer to execute the pipeline if too many long-running tests are combined in a single image test job. This might be a problem in the case of gemnasium-maven, gemnasium-python and retire.js because they build the project.

There might be leftover when building a project during the test, with possible side-effects. That's because the DockerRunner of the integration-test reuses the fixtures directory. TODO: create an issue about cleaning up the fixtures directory after running an RSpec group, or to make the runner copy the fixture to a temporary directory every time it runs.

Involved components

Optional: Intended side effects

See improvements

Optional: Missing test coverage

See risks

Testing

The integration tests provide coverage. SET may need to refactor, replace or remove the Test Orchestrator.

Edited by Will Meek