Skip to content

Image integration tests

Fabien Catteau requested to merge 330093-image-tests into master

What does this MR do?

Introduce image integration tests:

  • Add RSpec examples that check the Docker image of the analyzer
  • Add a new image test job that run these RSpec examples

RSpec examples

The RSpec examples check the behavior of the image when scanning supported test projects, repeating some of the existing job integration tests. What is checked:

  • report matches the expectation
  • report validates its JSON schema
  • exit code is 0

In addition, there are RSpec examples that check the behavior of the image when:

  • there's no supported dependency file
  • Gemfile.lock is renamed to gems.locked
  • go.sum is moved to a sub-directory
  • composer.lock isn't a valid JSON file

The TestProjectScan helper class is used to clone or reset a test project, and scan it using the Docker image.

The analyzer is executed in before(:context) blocks so that the same scan can be reused across multiple specs/examples. Unfortunately it's not possible to use let with before(:context).

image test job

The job uses a Docker image that supports Docker-in-Docker so that docker run can be executed.

The before_script installs the tools needed to run rspec:

  • ruby, bundler, and all the dependencies listed in the Gemfile
  • ruby-json so that created security reports can be parsed and checked
  • ruby-bigdecimal, a dependency of the json-validation Ruby gem
  • git, in order to clone the test projects

Follow-up MRs

  • Update developer docs and explain how to run image integration tests locally, in the project README
  • Enable rubocop and adjust its rules; see comment
  • Move job definition to analyzer.yml, the CI config shared by Secure analyzers implemented in Go
  • Port all job integration tests, except the ones for the offline environment maybe

Also, we might consider these:

  • Share RSpec helpers using a Ruby gem
  • Create a YAML file that lists all the integration tests
    • Iterate this list to create image integration tests, using RSpec
    • Iterate this list to create a CI config with job integration tests

What are the relevant issue numbers?

gitlab-org/gitlab#330093 (closed)

Does this MR meet the acceptance criteria?

Edited by Fabien Catteau

Merge request reports