Technical discovery: how to reduce reliance upon downstream QA projects in Static Analysis

Release notes

Problem to solve

The primary method by which changes are validated in the Category:SAST and Category:Secret Detection is via downstream QA pipelines. While this provides a nice integration test layer with the larger GitLab rails platform, it causes a couple significant problems:

• Testing the generated output of analyzers cannot be reliably done locally.
  • Generated output is influenced by default configuration of SAST or Secret Detection, as defined by the vendored templates.
• Users kicking off downstream QA pipelines must be maintainers of the test projects, too.

The integration tests we have today are necessary for final delivery of a change to the analyzer, but they are in the way of both contributions outside of the Static Analysis group as well as reliably testing changes locally. We need a better testing strategy which enables both.

Questions to explore

1. How can existing downstream QA projects be moved into the analyzers?
2. Which, if any, downstream QA projects do we have which should remain as they are?
3. What is the proper place for downstream QA projects in the Static Analysis CI/CD pipelines?

Constraints

• Jobs in the CI pipelines uses by projects in product for Static Analysis must be fully executable by everyone, including those who are not part of @gitlab-org/secure/static-analysis-be.
• Tests utilized in CI pipelines must be executable on local workstations and not wholly dependent upon GitLab.com CI pipelines.

Proposal

Migrate actively-maintained groupstatic analysis analyzers to use integration-test framework. See PoC for semgrep