Skip to content

Migrate downstreams to `integration-test`

James Liu requested to merge jliu-integration-test into main

What does this MR do?

Replaces downstream tests with integration-test by:

  • copying each downstream test repository into a subfolder of qa/fixtures/
  • adding an RSpec file that follows the patterns established by mobsf!60 (merged) and semgrep!146 (diffs)
  • adding separate QA expectations and tests for runs with and without GITLAB_FEATURES="sast_fp_reduction, which causes the primary_identifiers array to be populated.
  • upgrading the report package to v3.17.0 as sorting logic is gradually being moved from integration-test into the report package. Upgrading the package helps us achieve get closer to a deterministic sort order for the vulnerabilities array in the generated report.
  • updating the existing expectations under qa/expectations with the actual reports produced by the integration-test runner. I've verified that there are no functional changes besides differences in ordering, the addition of blank fields (like tracking and links), and new timestamps for the report creation date. This was done by running diff -u <(jq -S '.vulnerabilities | map(.identifiers | first) | sort' "<ACTUAL>") <(jq -S '.vulnerabilities | map(.identifiers | first) | sort' "<EXPECTED>")
  • adding logic to the spec to explicitly sort the actual and expected reports for ansible and terraform (see why below).

I also noticed something odd with some expectations:

  • For Ansible, we have two findings where the description is the only unique field, meaning both have the same cve value. This is caused by this Kics rule which can potentially match multiple policies against the same object.
  • For Terraform, we also have two findings with the same cve value. This occurs because both findings occur on the same line and column number (but different files), however we don't use the filename to compute the cve value.

These issues are troublesome because:

  1. The tests aren't deterministic anymore because the sorting logic in report uses the cve value. This will cause flakiness in the integration-test jobs.
  2. We'll run into more issues related to gitlab-org/gitlab#374496 (closed)

We can address 2. with a follow-up issue.

What are the relevant issue numbers?

Does this MR meet the acceptance criteria?

Edited by Zach Rice

Merge request reports