Replace job tests with image tests in gemnasium-maven
Summary
Update gemnasium-maven to replace job integration tests (using external test projects and running in separate downstream pipelines) with image integration tests (using rspec and all running in the same image test
job).
Implementation plan
-
Add rspec to the project:
- add a Gemfile
- create a
spec
directory - create an
spec/image_spec.rb
file - add an image test job
-
For each of the following downstream QA job integration tests in the gemnasium project:
-
ds-excluded-paths-qa -
java-maven-qa -
java-maven-java-8-qa -
java-maven-java-11-qa -
java-maven-java-13-qa -
java-maven-java-14-qa -
java-maven-java-15-qa -
java-maven-java-16-qa -
java-maven-offline-qa -
java-maven-multimodule-qa -
java-gradle-qa -
java-gradle-as-maven-qa -
java-gradle-8-qa -
java-gradle-11-qa -
java-gradle-13-qa -
java-gradle-14-qa -
java-gradle-15-qa -
java-gradle-16-qa -
java-gradle-kotlin-dsl-qa -
scala-sbt-qa -
scala-sbt-java-8-qa -
scala-sbt-java-11-qa -
java-gradle-multimodule-qa -
java-gradle-no-root-dependencies-qa -
scala-sbt-1-0 -
scala-sbt-1-1 -
scala-sbt-1-2 -
scala-sbt-1-3 -
scala-sbt-1-4 -
scala-sbt-multiproject-qa
Perform the following tasks:
- check whether the job integration test can be replaced by an rspec test
- if yes, copy the files from the test project to the
qa/fixtures
directory ofgemnasium
; the sub-directory name istest_project_name/test_case_name
, and the default test case is "main" - add an RSpec example to the
spec/image_spec.rb
file that replicates what's tested in the downstream pipeline - make sure that the new RSpec examples are executed when running the
image test
job - remove the downstream pipeline from the CI config
-
Also, we'll need to make sure to keep at least one job integration test, to check the Dependency Scanning template.
Image integration tests are implemented using the integration-test project. See gemnasium's spec/image_spec.rb.
Improvements
- Image integration tests can run locally.
- They are easier to maintained, compared to job integration tests using external test projects.
- They run faster because there's no need to spin off a new job for each test case.
Risks
We might miss a case where the job definition can't be ignored. For instance, the image test might not set all the environment variables set in the scanning job. See #330093 (closed).
It might take longer to execute the pipeline if too many long-running tests are combined in a single image test
job. This might be a problem in the case of gemnasium-maven, gemnasium-python and retire.js because they build the project.
There might be leftover when building a project during the test, with possible side-effects. That's because the DockerRunner
of the integration-test reuses the fixtures directory. TODO: create an issue about cleaning up the fixtures directory after running an RSpec group, or to make the runner copy the fixture to a temporary directory every time it runs.
Optional: Intended side effects
See improvements
Optional: Missing test coverage
See risks
Testing
The integration tests provide coverage. SET may need to refactor, replace or remove the Test Orchestrator.