Skip to content

Add ability for image integration test to regenerate expectation files

Proposal

If a project uses image integration tests and a change is made to the contents of a generated report, then a large number of expectation files need to be updated. For example, this gemnasium MR required updating 42 expectation files by hand.

The purpose of this issue is to provide some method, perhaps a flag, that can be passed when running image integration tests, to automatically generate (and possibly overwrite) the expectation files.

Implementation Plan

  1. Implement Add changelog and versioning to integration tes... (#382460 - closed) • Adam Cohen • 18.3 so we can make breaking changes.

  2. Add logic to check for a REFRESH_EXPECTED environment variable to the following shared examples:

  3. Replace the recorded_report variable with expectation_path, which contains the path to the expectation file.

  4. Update scripts/*-qa.rb to use the new expectation_path variable.

  5. Add instructions to the readme explaining how to migrate from previous versions of the integration-test image, and how to refresh expectations.

How to refresh expectations

  1. Migrate from the stable image to v2

  2. Follow the directions for Refreshing Expectations:

    Pass REFRESH_EXPECTED=true when running the integration test:

    docker run -it --rm -v "$PWD:$PWD" -w "$PWD" \
      -e TMP_IMAGE=kics:main \
      -e REFRESH_EXPECTED=true \
      -v /var/run/docker.sock:/var/run/docker.sock \
      registry.gitlab.com/gitlab-org/security-products/analyzers/integration-test:2 rspec spec/kics_image_spec.rb

/cc @craigmsmith @fcatteau @hacks4oats

Edited by Adam Cohen