Technical discovery: how to reduce reliance upon downstream QA projects in Static Analysis
Release notes
Problem to solve
The primary method by which changes are validated in the Category:SAST and Category:Secret Detection is via downstream QA pipelines. While this provides a nice integration test layer with the larger GitLab rails platform, it causes a couple significant problems:
- Testing the generated output of analyzers cannot be reliably done locally.
- Generated output is influenced by default configuration of SAST or Secret Detection, as defined by the vendored templates.
- Users kicking off downstream QA pipelines must be maintainers of the test projects, too.
The integration tests we have today are necessary for final delivery of a change to the analyzer, but they are in the way of both contributions outside of the Static Analysis group as well as reliably testing changes locally. We need a better testing strategy which enables both.
Questions to explore
- How can existing downstream QA projects be moved into the analyzers?
- Which, if any, downstream QA projects do we have which should remain as they are?
- What is the proper place for downstream QA projects in the Static Analysis CI/CD pipelines?
Constraints
- Jobs in the CI pipelines uses by projects in product for Static Analysis must be fully executable by everyone, including those who are not part of @gitlab-org/secure/static-analysis-be.
- Tests utilized in CI pipelines must be executable on local workstations and not wholly dependent upon GitLab.com CI pipelines.
Proposal
Migrate actively-maintained groupstatic analysis analyzers to use integration-test
framework. See PoC for semgrep
Edited by Lucas Charles