Unstable vulnerability ordering on security reports
Our test projects within security-products/tests occasionally need their fixtures (qa/expect/gl-sast-report.json) updated to reflect changes in the reports, usually due to new advisories in external DBs. There are however some false-negatives when report diffing fails due to ordering issues. While the
sast analyzer should consistently be sorting reports there is an issue to be investigated.
Previous occurrences of report changes:
Previous discussion on ordering:
Broken pipeline due to report changes: https://gitlab.com/gitlab-org/security-products/tests/js-yarn/pipelines/43743205
What is the current bug behavior?
Test projects occasionally generate reports with a different vulnerability order than fixtures, but no changes in set of identified vulnerabilities.
What is the expected correct behavior?
- Reports should have a consistent order of vulnerabilities.
- Test projects should only fail on differences between vulnerabilities within reports, not differences in report order.
Review common lib sorting/deduping logic to ensure order is consistent across reports.
If it's not possible to guarantee a consistent order, our pipeline diffing should sort reports itself, but this is not ideal.