Skip to content

Update integration-test project to skip fields when comparing reports

Summary

Today job integration tests do a strict comparison of generated reports with expected ones, and this strict comparison create maintenance issues:

  • Some test projects like tests/python-pip don't have a lock file, and they depend on package that might get new versions at any time. As a result, the generated Dependency Scanning report might change, and the expected report needs to be updated. (Same for the SBOMs but these are not compared at the moment.)

Proposal

Change the QA scripts so that they pass when the versions of the project dependencies don't match the expected report.

(Ideally, the tests would only pass when package versions are higher that expected, but comparing versions might be difficult because of the different package managers we need to cover.)

Improvements

  • Pipelines of the test projects become very robust.
  • They don't break because of minor changes to the analyzer projects.
  • They don't repeat the image integration tests.

Involved components

The change impacts all the test projects for the Dependency Scanning analyzers. However, these projects don't need to change.

Implementation Plan

  1. Update the comparable module to ignore a field if it equals :SKIP:.

  2. Ensure that the CycloneDX shared examples will also ignore fields containing :SKIP:.

  3. Update the report_shared_examples.rb to use include_json instead of eql.

  4. Add unit tests to the integration-test project to verify that fields marked :SKIP: are ignored.

  5. Remove sorting from the GitlabSecure::IntegrationTest::Comparable module.

  6. Update the Report.Sort() function of the report package to sort Vulnerabilities by Severity, CompareKey, and Location.Dependency.Version (if available)

  7. Update the expectation files in the gemnasium project to mark some of the vulnerability/dependency fields as :SKIP:. The following MRs have updated expectations fields that have changed as a result of moving dependency targets and can be used as a starting point.

  8. Update the expectations in gemnasium that have changed after implementing steps 5 and 6 above.

  9. Add a job that checks if a field marked as :SKIP: is removed. The job should include allow_failure: true so that it does not block MR progress, but does alert the reviewer(s).

    The following script that can be used to check for this as a first iteration:

    # Check if the proposed changes remove a field with value ":SKIP:".
DIFF_CHECK=$(git diff $CI_MERGE_REQUEST_TARGET_BRANCH_SHA $CI_MERGE_REQUEST_SOURCE_BRANCH_SHA | egrep '^-[^"]*"[^"]+":\s?":SKIP:"')
if [[ $DIFF_CHECK != "" ]]; then
    echo "A previously skipped dependency or component field has been modified."
    echo "Please review these changes since this has caused test targets to continuously move."
    exit 1;
fi

    Ignore fields marked with `:SKIP:` when compari... (gitlab-org/security-products/analyzers/gemnasium!444 - merged)

Edited by Adam Cohen