Migrate downstreams to `integration-test`
What does this MR do?
Replaces downstream tests with integration-test
by:
- copying each downstream test repository into a subfolder of
qa/fixtures/
- adding an RSpec file that follows the patterns established by mobsf!60 (merged) and semgrep!146 (diffs)
- adding separate QA expectations and tests for runs with and without
GITLAB_FEATURES="sast_fp_reduction
, which causes theprimary_identifiers
array to be populated. - upgrading the
report
package to v3.17.0 as sorting logic is gradually being moved fromintegration-test
into the report package. Upgrading the package helps usachieveget closer to a deterministic sort order for thevulnerabilities
array in the generated report. - updating the existing expectations under
qa/expectations
with the actual reports produced by theintegration-test
runner. I've verified that there are no functional changes besides differences in ordering, the addition of blank fields (liketracking
andlinks
), and new timestamps for the report creation date. This was done by runningdiff -u <(jq -S '.vulnerabilities | map(.identifiers | first) | sort' "<ACTUAL>") <(jq -S '.vulnerabilities | map(.identifiers | first) | sort' "<EXPECTED>")
- adding logic to the spec to explicitly sort the actual and expected reports for
ansible
andterraform
(see why below).
I also noticed something odd with some expectations:
- For Ansible, we have two findings where the
description
is the only unique field, meaning both have the samecve
value. This is caused by this Kics rule which can potentially match multiple policies against the same object. - For Terraform, we also have two findings with the same
cve
value. This occurs because both findings occur on the same line and column number (but different files), however we don't use the filename to compute thecve
value.
These issues are troublesome because:
The tests aren't deterministic anymore because the sorting logic in report uses thecve
value. This will cause flakiness in theintegration-test
jobs.- We'll run into more issues related to gitlab-org/gitlab#374496 (closed)
We can address 2. with a follow-up issue.
What are the relevant issue numbers?
Does this MR meet the acceptance criteria?
-
Changelog entry added -
Documentation created/updated for GitLab EE https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/10527 -
Documentation created/updated for this project, if necessary -
Documentation reviewed by technical writer or follow-up review issue created -
Tests added for this feature/bug -
Job definition updated, if necessary -
Conforms to the code review guidelines -
Conforms to the Go guidelines -
Security reports checked/validated by reviewer
Edited by Zach Rice