Add a test or review step that explicitly checks the set of rules that are active in the Semgrep-based SAST analyzer
Background
As discussed in various issues under Ensure all rules from SAST-rules are successful... (&13973), but especially Semgrep-based analyzer image omits files that a... (#463397 - closed), we have unintentionally omitted some rules from the shipped image.
The idea behind this issue is to detect during the analyzer-update process if such a problem is happening again for whatever reason.
Proposal
Add a test or manual process to the Semgrep-based analyzer that:
- Uses the existing capability where the report includes the list of rules it has used for scanning.
- Surfaces the fact that no rule identifiers have changed.
This could be automated (for example by finding all primary identifiers from a sast-rules
dist
package) or manual (for example as a review step whenever updating the sast-rules version: "Did you expect new rules? Nothing changed... maybe check that?").
qa/expect/c/with-primary-identifiers/gl-sast-report.json
appears to be the only test case that uses this feature. See gitlab-org/security-products/analyzers/semgrep!424 (diffs) where the MR diff demonstrates that this file's contents changes when new rules are added.