Skip to content

Improve SAST analyzer tests

Summary

The current QA process for analysers maintained by groupstatic analysis involves executing the analyser against a corresponding test project via a downstream pipeline. We'll use the flawfinder analyser and the C test project in this document going forward.

This is a simplified overview of the process:

  1. A change to the analyser is pushed and results in a new Docker image being built.
  2. The CI configuration invokes the CI of the test project using a downstream pipeline via a trigger.
  3. The CI pipeline of the test project is executed against the new analyser image.
  4. After a successful run, the expected and actual gl-sast-report.json files are compared.
  5. The downstream pipeline is marked as successful or failed accordingly.

While the nature of test projects means that analysers are being executed in a production-like environment (all CI variables exist, the SAST template is used etc.), they pose a few significant problems. These are listed in no particular order:

  1. Community contributors are unable to validate their changes independently, as they're unable to run the full pipeline. See gitlab-org/security-products/analyzers/semgrep!121 (closed) and gitlab-org/security-products/analyzers/security-code-scan!122 (merged) for examples where pipelines don't run for community contributions.
  2. Changes to test projects can have unintended consequences for non-SAST analysers.
  3. Changes to test projects can be hampered by circular dependencies/changes to test projects must be orchestrated with the corresponding analyser. For example, to address #223283 we need to change folder names within the test project and the qa/expect fixtures in the analyser. This results in two MRs which will cause pipeline failures for each other.
  4. The -FREEZE branches can be difficult to grok. While there are only two for the C test project, some have more. Compare the branch view with the directory layout in Gemnasium; the latter is much easier to understand.
  5. Difficult to run tests locally. We have an analyzer-test-downstreams script that simplifies the process, but this is maintained separately to the analysers and not exercised frequently, so issues can easily arise.

There are more problems which were better articulated and discussed in #297361 (closed) which I won't repeat here.

Proposal

Do what groupcomposition analysis has done for Gemnasium. This involves the following (I've included a link to the original issue where possible):

  1. Copying test projects to qa/fixtures subfolders inside the analyser repositories. #297361 (closed)
  2. Investigating if the existing RSpec integration-test tooling works for SAST analysers (they should).
  3. Setting up dynamic child pipelines to execute the analyser against each relevant test project inside qa/fixtures. This is referred to as an image test. #336330 and #336322 (closed)
  4. Removing downstream pipelines from the analyser's CI configuration. The test projects will still be executed by the Secure Test Orchestrator on a regular basis as a production-like test.
  5. Simplifying the downstream test projects to exercise the integration with GitLab CI rather than asserting on the gl-sast-report.json files. #336332 (closed)
  6. For FIPS enablement, all downstreams should work as-expected: add a dedicated integration-test-fips job that runs all semgrep and kics jobs overriding the image to use the IMAGE_PREFIX. See this job for last attempt (note failure case and linked commit for workable job) https://gitlab.com/gitlab-org/security-products/analyzers/semgrep/-/jobs/3233680743#L115

Scope

groupstatic analysis owns many analysers that'll need to be updated, including those for Secret Detection and Code Quality. I'd like to leave those out of scope as:

We also have deprecated analysers (bandit, eslint, and gosec) which should also be descoped.

That leaves the remaining analysers in-scope for this change

Future work

Explore the possibility of end-to-end testing within the GitLab QA process. @fcatteau we talked briefly about this the other week, do you mind adding some detail on what this looks like?

Implementation plan

Edited by Vishwa Bhat