Update Secure common lib to provide ignorelist for supporting complex file exclusions
Problem to solve
We recently added file exclusion support for sast and dependency scanning using the
EXCLUDED_PATHS env vars. This is useful in providing a generic approach to excluding results but can be difficult to map to individual scanners since not all provide an exclusion option to propagate and many that do are not necessarily compatible with our exclusion strategy (i.e. regex format, blob types, exact matches, etc). To fix this we decided to filter vulnerabilities in the orchestrator after the scan has occurred, which works generically for all scanners but requires a full scan regardless.
Alternatively, we should look at adding support to the common library for generating an
EXCLUDED_PATHS. This can be propagated to those scanners that require exact file matches, such as
pmd-apex, and prevent unnecessary scans of all project files. As stated by @fcatteau
we could build the list using https://gitlab.com/gitlab-org/security-products/analyzers/common/blob/master/pathfilter/match.go#L24 and https://golang.org/pkg/path/filepath/#Walk, and then pass it to whatever CLI can process such list, but passing SAST_EXCLUDED_PATHS to find won't work - the syntax is not compatible
Add a new method to
pathfilter which returns a list of the absolute paths of all files to be ignored. This can be optionally called by any relevant analyzer to perform a more effective scan.
Permissions and Security
This would result in no user-facing change as it would involve internally parsing
*_EXCLUDED_PATHS more effectively.
We should test against a project in which the analyzer will fail if scanning a directory explicitly listed within
EXCLUDED_PATHS; i.e. a file with a syntax error.
What does success look like, and how can we measure that?
More performant analyzers, analyzers that can correctly skip code that should not be scanned.