Add ability for image integration test to regenerate expectation files
Proposal
If a project uses image integration tests and a change is made to the contents of a generated report, then a large number of expectation files need to be updated. For example, this gemnasium MR required updating 42 expectation files by hand.
The purpose of this issue is to provide some method, perhaps a flag, that can be passed when running image integration tests, to automatically generate (and possibly overwrite) the expectation files.
Implementation Plan
-
Implement Add changelog and versioning to integration tes... (#382460 - closed) • Adam Cohen • 18.3 so we can make breaking changes. -
Add logic to check for a REFRESH_EXPECTEDenvironment variable to the following shared examples: -
Replace the recorded_report variable with expectation_path, which contains the path to the expectation file. -
Update scripts/*-qa.rbto use the newexpectation_pathvariable. -
Add instructions to the readme explaining how to migrate from previous versions of the integration-testimage, and how to refresh expectations.
How to refresh expectations
-
Follow the directions for Refreshing Expectations:
Pass
REFRESH_EXPECTED=truewhen running the integration test:docker run -it --rm -v "$PWD:$PWD" -w "$PWD" \ -e TMP_IMAGE=kics:main \ -e REFRESH_EXPECTED=true \ -v /var/run/docker.sock:/var/run/docker.sock \ registry.gitlab.com/gitlab-org/security-products/analyzers/integration-test:2 rspec spec/kics_image_spec.rb