Perform a sort-independent comparison with expectations in Security Products QA tests
Problem to solve
As explained in https://gitlab.com/gitlab-org/gitlab-ee/issues/9452 we have unstable sorting of vulnerabilities in our security reports. This is annoying and causes flaky test in our QA projects: https://gitlab.com/gitlab-org/security-products/tests
Intended users
~Secure people
Further details
As we're moving logic to the backend and removing orchestrator layer, sorting will ultimately happen on the rails backend anyway, where reports from multiple jobs will be aggregated.
This means there is no longer user need to invest time on producing a stable order right from the report generation, which also implies some challenges as each report type might need a property specific sort.
Proposal
Update our QA templates to do a sort independent comparison like using jq tool instead of the good old plain diff
command.
Example comparison
jq --argfile a qa/expect/gl-sast-report.json --argfile b ./gl-sast-report.json -n '($a | (.. | arrays) |= sort) as $a | ($b | (.. | arrays) |= sort) as $b | $a == $b'
Documentation
I don't think there is any documentation to update regarding this.
What does success look like, and how can we measure that?
No more flaky tests because of the order of items in the vulnerabilities array.