Skip to content

Support double star globs in SAST and Secret Detection excluded paths

⚠ Dependency Scanning has been split out to #368052 (closed).

Problem to solve

Follow-up of !35037 (comment 367455440). Users should be able to use double star globs in the patterns provided. As we can see in the MR and related QA tests, patterns like **/test/* are accepted without any warning, but won't work as what we can expect (all folders named test in the current tree).

Starting from %13.0, the CI config supports globs and double starts in cache:paths using doublestar.Glob. The path filters for SAST and Dependency Scanning could also use doublestar.Glob to support globs, and to improve consistency. Also, this would reduce maintenance cost, and the need for documentation.

Intended users

Proposal

Use doublestar.Glob in the pathfilter package of the analyzer common library. The unit tests need to be updated but the interface remains the same, so the analyzer projects won't change. We should maintain backwards compatibility with the existing behaviour of pathfilter.Match (namely the support around gitignore-style matching).

Alternatively, we should better document the existing behavior, and the discrepancy b/w the patterns supported by catche:paths and the *_EXCLUDED_PATHS variables.

Implementation plan

  1. Introduce the doublestar package to the common library. An MR is in progress.
    • Add tests containing globstar (**) patterns to match_test.go.
    • Modify the Match function to match patterns using doublestar.Match instead of filepath.Match under certain conditions (globstar patterns should not be truncated).
    • Release a new minor version of common.
  2. Bump the command package to use the new version of common. Analysers filter paths via this package.
  3. Bump the command (and also common, might as well) package in the following analysers and release a new minor version of each. Note that many analysers are using v1 or v2 of common, so in addition to go.mod the import paths will need to be updated to v3. The main change in v3 is removing the deprecated command and issue modules from the source. All analysers should already have been updated to use the replacements, so there should be minimal change updating to v3 of common.
  4. Modify the SAST and DS documentation. A draft MR has been prepared.

Permissions and Security

N/A

Documentation

Update SAST docs and Dependency Scanning docs.

Availability & Testing

Currently, SAST_EXCLUDED_PATHS and DS_EXCLUDED_PATHS are not covered by QA. We might want to address that behavior switching to doublestar, and possibly changing the behavior. Or, we can considered that unit tests are sufficient, in the common library.

What is the type of buyer?

GitLab Ultimate

Is this a cross-stage feature?

Yes, it applies to all analyzers maintained by devopssecure, except Category:DAST.

Links / references

!35037 (comment 367455440)

Edited by Connor Gilbert