Improve SAST analyzer tests
Summary
The current QA process for analysers maintained by groupstatic analysis involves executing the analyser against a corresponding test project via a downstream pipeline. We'll use the flawfinder analyser and the C test project in this document going forward.
This is a simplified overview of the process:
- A change to the analyser is pushed and results in a new Docker image being built.
- The CI configuration invokes the CI of the test project using a downstream pipeline via a trigger.
- The CI pipeline of the test project is executed against the new analyser image.
- After a successful run, the expected and actual
gl-sast-report.json
files are compared. - The downstream pipeline is marked as successful or failed accordingly.
While the nature of test projects means that analysers are being executed in a production-like environment (all CI variables exist, the SAST template is used etc.), they pose a few significant problems. These are listed in no particular order:
- Community contributors are unable to validate their changes independently, as they're unable to run the full pipeline. See gitlab-org/security-products/analyzers/semgrep!121 (closed) and gitlab-org/security-products/analyzers/security-code-scan!122 (merged) for examples where pipelines don't run for community contributions.
- Changes to test projects can have unintended consequences for non-SAST analysers.
- Changes to test projects can be hampered by circular dependencies/changes to test projects must be orchestrated with the corresponding analyser. For example, to address #223283 we need to change folder names within the test project and the
qa/expect
fixtures in the analyser. This results in two MRs which will cause pipeline failures for each other. - The
-FREEZE
branches can be difficult to grok. While there are only two for the C test project, some have more. Compare the branch view with the directory layout in Gemnasium; the latter is much easier to understand. - Difficult to run tests locally. We have an analyzer-test-downstreams script that simplifies the process, but this is maintained separately to the analysers and not exercised frequently, so issues can easily arise.
There are more problems which were better articulated and discussed in #297361 (closed) which I won't repeat here.
Proposal
Do what groupcomposition analysis has done for Gemnasium. This involves the following (I've included a link to the original issue where possible):
- Copying test projects to
qa/fixtures
subfolders inside the analyser repositories. #297361 (closed) - Investigating if the existing RSpec integration-test tooling works for SAST analysers (they should).
- Setting up dynamic child pipelines to execute the analyser against each relevant test project inside
qa/fixtures
. This is referred to as an image test. #336330 and #336322 (closed) - Removing downstream pipelines from the analyser's CI configuration. The test projects will still be executed by the Secure Test Orchestrator on a regular basis as a production-like test.
- Simplifying the downstream test projects to exercise the integration with GitLab CI rather than asserting on the
gl-sast-report.json
files. #336332 (closed) - For FIPS enablement, all downstreams should work as-expected: add a dedicated
integration-test-fips
job that runs allsemgrep
andkics
jobs overriding the image to use the IMAGE_PREFIX. See this job for last attempt (note failure case and linked commit for workable job) https://gitlab.com/gitlab-org/security-products/analyzers/semgrep/-/jobs/3233680743#L115
Scope
groupstatic analysis owns many analysers that'll need to be updated, including those for Secret Detection and Code Quality. I'd like to leave those out of scope as:
- Secret Detection relies on Git context that can't easily be managed in the analyser. See gitlab-org/secure/tools/analyzer-scripts!13 (comment 1009571174) for more information.
- Code Quality appears to follow a completely different structure to the rest of the analysers.
We also have deprecated analysers (bandit, eslint, and gosec) which should also be descoped.
That leaves the remaining analysers in-scope for this change
Future work
Explore the possibility of end-to-end testing within the GitLab QA process. @fcatteau we talked briefly about this the other week, do you mind adding some detail on what this looks like?
Implementation plan
-
Semgrep -
Semgrep PoC covering golang
andc
downstreams gitlab-org/security-products/analyzers/semgrep!157 (merged) -
Follow-up MR for stabilizing golang projects and reducing duplication gitlab-org/security-products/analyzers/semgrep!171 (merged) -
Semgrep js projects @vbhat161 | gitlab-org/security-products/analyzers/semgrep!195 (merged) -
Semgrep python projects @vbhat161 | gitlab-org/security-products/analyzers/semgrep!185 (merged) -
Semgrep java projects @vbhat161 | gitlab-org/security-products/analyzers/semgrep!181 (merged) -
Semgrep csharp projects @vbhat161 | gitlab-org/security-products/analyzers/semgrep!205 (merged) -
Semgrep FIPS downstreams (blocked on #381209 (closed))
-
-
Kics downstream projects : @jamesliu-gitlab - gitlab-org/security-products/analyzers/kics!62 (merged) -
Kics downstream projects -
Kics FIPS downstreams (blocked on #381209 (closed))
-
-
brakeman downstream projects: @vbhat161 | gitlab-org/security-products/analyzers/brakeman!114 (merged) -
kubesec downstream projects : @arpitgogia | gitlab-org/security-products/analyzers/kubesec!79 (merged) -
mobsf downstream projects @vbhat161 | gitlab-org/security-products/analyzers/mobsf!60 (merged) -
phpcs-security-audit downstream projects @zrice | gitlab-org/security-products/analyzers/phpcs-security-audit!77 (merged) -
pmd-apex downstream projects : @rossfuhrman - gitlab-org/security-products/analyzers/pmd-apex!97 (merged) and gitlab-org/security-products/analyzers/pmd-apex!99 (merged) -
security-code-scan downstream projects @zrice | gitlab-org/security-products/analyzers/security-code-scan!130 (merged) -
sobelow downstream projects : @rossfuhrman - gitlab-org/security-products/analyzers/sobelow!91 (merged) -
spotbugs downstream projects: @vbhat161 | gitlab-org/security-products/analyzers/spotbugs!157 (merged) -
secrets downstream projects: @theoretick | gitlab-org/security-products/analyzers/secrets!199 (merged)