:warning: Dependency Scanning has been split out to https://gitlab.com/gitlab-org/gitlab/-/issues/368052. ### Problem to solve Follow-up of https://gitlab.com/gitlab-org/gitlab/-/merge_requests/35037#note_367455440. Users should be able to use double star globs in the patterns provided. As we can see in the MR and related QA tests, patterns like `**/test/*` are accepted without any warning, but won't work as what we can expect (all folders named `test` in the current tree). Starting from %13.0, the CI config supports globs and double starts in [cache:paths](https://docs.gitlab.com/ee/ci/yaml/#cachepaths) using [doublestar.Glob](https://pkg.go.dev/github.com/bmatcuk/doublestar@v1.2.2?tab=doc#Match). The path filters for SAST and Dependency Scanning could also use [doublestar.Glob](https://pkg.go.dev/github.com/bmatcuk/doublestar@v1.2.2?tab=doc#Match) to support globs, and to improve consistency. Also, this would reduce maintenance cost, and the need for documentation. ### Intended users * [Delaney (Development Team Lead)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#delaney-development-team-lead) * [Sasha (Software Developer)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#sasha-software-developer) * [Sam (Security Analyst)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#sam-security-analyst) ### Proposal Use [doublestar.Glob](https://pkg.go.dev/github.com/bmatcuk/doublestar@v1.2.2?tab=doc#Match) in the [pathfilter](https://gitlab.com/gitlab-org/security-products/analyzers/common/-/tree/v2.10.3/pathfilter) package of the analyzer [common](https://gitlab.com/gitlab-org/security-products/analyzers/common) library. The unit tests need to be updated but the interface remains the same, so the analyzer projects won't change. We should maintain backwards compatibility with the existing behaviour of `pathfilter.Match` (namely the support around gitignore-style matching). Alternatively, we should better document the existing behavior, and the discrepancy b/w the patterns supported by `catche:paths` and the `*_EXCLUDED_PATHS` variables. ### Implementation plan 1. Introduce the [doublestar](https://github.com/bmatcuk/doublestar) package to the [common](https://gitlab.com/gitlab-org/security-products/analyzers/common) library. An [MR](https://gitlab.com/gitlab-org/security-products/analyzers/common/-/merge_requests/162) is in progress. - Add tests containing globstar (**) patterns to [match_test.go](https://gitlab.com/gitlab-org/security-products/analyzers/common/-/blob/master/pathfilter/match_test.go). - Modify the [Match](https://gitlab.com/gitlab-org/security-products/analyzers/common/-/blob/330a662dbd14332e7e2bf5d13060dc8d5e99be38/pathfilter/match.go#L24) function to match patterns using `doublestar.Match` instead of `filepath.Match` under certain conditions (globstar patterns should not be truncated). - Release a new minor version of `common`. 1. Bump the [`command`](https://gitlab.com/gitlab-org/security-products/analyzers/command) package to use the new version of `common`. Analysers filter paths via this package. 1. Bump the `command` (and also `common`, might as well) package in the following analysers and release a new minor version of each. Note that many analysers are using v1 or v2 of `common`, so in addition to `go.mod` the import paths will need to be updated to `v3`. The main change in `v3` is [removing the deprecated `command` and `issue` modules](https://gitlab.com/gitlab-org/security-products/analyzers/common/-/merge_requests/158) from the source. All analysers should already have been updated to use the replacements, so there should be minimal change updating to `v3` of `common`. - Think about automating this process using a script. - [x] https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium - [x] https://gitlab.com/gitlab-org/security-products/analyzers/brakeman - [x] https://gitlab.com/gitlab-org/security-products/analyzers/security-code-scan - [x] https://gitlab.com/gitlab-org/security-products/analyzers/semgrep - [x] https://gitlab.com/gitlab-org/security-products/analyzers/mobsf - [x] https://gitlab.com/gitlab-org/security-products/analyzers/flawfinder - [x] https://gitlab.com/gitlab-org/security-products/analyzers/kics - [x] https://gitlab.com/gitlab-org/security-products/analyzers/sobelow - [x] https://gitlab.com/gitlab-org/security-products/analyzers/phpcs-security-audit - [x] https://gitlab.com/gitlab-org/security-products/analyzers/secrets - [x] https://gitlab.com/gitlab-org/security-products/analyzers/kubesec - [x] https://gitlab.com/gitlab-org/security-products/analyzers/nodejs-scan - [x] https://gitlab.com/gitlab-org/security-products/analyzers/pmd-apex - [x] https://gitlab.com/gitlab-org/security-products/analyzers/spotbugs 1. Modify the SAST and DS documentation. A [draft MR](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91338) has been prepared. ### Permissions and Security N/A ### Documentation Update [SAST docs](https://docs.gitlab.com/ee/user/application_security/sast/index.html) and [Dependency Scanning docs](https://docs.gitlab.com/ee/user/application_security/dependency_scanning/index.html). ### Availability & Testing Currently, `SAST_EXCLUDED_PATHS` and `DS_EXCLUDED_PATHS` are not covered by QA. We might want to address that behavior switching to [doublestar](https://pkg.go.dev/github.com/bmatcuk/doublestar@v1.2.2?tab=doc#Match), and possibly changing the behavior. Or, we can considered that unit tests are sufficient, in the `common` library. ### What is the type of buyer? ~"GitLab Ultimate" ### Is this a cross-stage feature? Yes, it applies to all analyzers maintained by ~"devops::secure", except ~"Category:DAST". ### Links / references https://gitlab.com/gitlab-org/gitlab/-/merge_requests/35037#note_367455440