Provide image integration test job and helpers to Secure analyzer projects
Problem to solve
As a developer working on a GitLab Secure analyzer project, I'd like to write specs for the analyzer image, but I can't reuse the RSpec shared examples used to test the gemnasium
image (Dependency Scanning). I need an easy way to set up image integration tests similar to the ones of the Gemnasium projects without additional maintenance cost.
User experience goal
As a developer of a GitLab Secure analyzer, I write RSpec examples to test the analyzer image. These examples use shared examples for image testing. I then set up a CI job that simply calls rspec
in the context of an image that provides the shared examples as well as the system dependencies needed to run these shared examples.
Proposal
- Make the shared examples generic enough to handle Dependency Scanning, Secret Detection, and SAST.
- Create a Ruby gem that combines the shared examples, the Ruby dependencies, and the Ruby classes developers commonly use to write image specs.
- Create a new project that builds an image where Ruby, rspec, and the aforementioned Ruby gem are installed. The image also provides the system libraries needed to run the specs (such as
ruby-json
), the Docker CLI, and the JSON schemas needed for schema validation. - Document this project, and explain how to use it to write image integration tests.
- Use this in the
gemnasium
project, ~"Category:Dependency Scanning". - Invite project maintainers of Category:SAST and Category:Secret Detection to use it to.
The project name is to be defined. See discussion.
Optional: Add the image test
job to the shared CI config, and set the rules:exists
parameter so that the job is skipped when there are no image integration tests.
Implementation plan
-
Update the gemnasium
project to make the shared examples generic-
Include all security report schemas, so that it can validate any report gitlab-org/security-products/analyzers/gemnasium!212 (merged)
-
-
Import shared examples and dependencies into a new integration-test project -
Make this new project build a Docker image that contains all we need for integration tests -
Use this Docker image in the gemnasium
project -
Update README of integration-test project and cover image integration tests using rspec gitlab-org/security-products/analyzers/integration-test!5 (merged) -
Share with other Secure teams, and encourage them to use the same testing tool => brown bag session on July 29th
Follow-up issues:
-
Move definition of image test
job to shared CI config (optional, if doesn't impact other Secure analyzers) -
Convert job integration tests to image integration tests when relevant -
Publish gitlab-secure_integration-test
Rubygem, so that it can be used out of the integration-test image
Further details
The gemnasium project has an image test
job that runs the specs defined in spec/image_spec.rb. These specs directly use generic code:
- shared examples for the generated report
- shared examples for the scan itself (exit code)
- a helper class that runs the analyzer image and collects the generated report as well as the exit code
This generic code could be used to test the Docker images of other analyzer projects.
Documentation
The new project should document:
- how to use the gem in rspec examples
- shared examples for reports
- shared examples for scans
- runner class
- how to check reports using the QA scripts
- how to use the Docker image
- job template for image integration tests
- job template for job integration tests
Availability & Testing
- The
Comparable
module is tested using rspec. It's used in shared examples for images. - Shared examples for image are tested when testing QA scripts.
- The QA scripts are tested by comparing expected report to similar and different reports.
- TBD: Should we test the shared examples for reports?
- TBD: Should we test the runner class?
What does success look like, and how can we measure that?
Developers working on Secure analyzer projects can easily set up image integration tests.
Is this a cross-stage feature?
Potentially. This could be used by devopsprotect.