Make Gemnasium skip files listed in DS_EXCLUDED_PATHS

Release notes

Variable DS_EXCLUDED_PATHS behaviour changed to pre-filter

For users of the Depedency Scanning variable DS_EXCLUDED_PATHS we are introducing a breaking change, it will now pre-filter. This change applied to all versions asthe change is being backported. December 22, 2021 forward gemnasium considers DS_EXCLUDED_PATHS when searching for supported projects and will pre-filter out those that match. Pre-filtering prevents the analyzer from logging warning or failing when processing dependency files that have been explicitly excluded using DS_EXCLUDED_PATHS.

You should not need to take any action, however if you were expecting this post-filtering behaviour you will no longer have this data available to you in logs or artifacts.

The previous behaviour was causing failures and unexpected errors for users and after discussions we found that this, pre-filter as opposed to post filter, was the more expected and desired behaviour.

https://docs.gitlab.com/ee/user/application_security/dependency_scanning/#configuring-dependency-scanning

Problem to solve

Gemnasium can be configure to filter specific files and directories using the DS_EXCLUDED_PATHS variable, but this is implemented as a post-filter, which causes issues:

when the excluded build file, dependency file, or lock file can't be processed
when the analyzer only scans one file/directory, and it automatically picks up the one being excluded

The workaround is to remove the files being excluded prior to the scan, in the before_script of the scanning job.

Intended users

User experience goal

Proposal

Make Gemnasium consider DS_EXCLUDED_PATHS when searching for supported projects, and skip files that match this CI file (pre-filtering). Pre-filtering would prevent the analyzer from logging warning or even failing when processing dependency files that have been explicitly excluded using DS_EXCLUDED_PATHS.

Further details

Gemnasium relies on the pathfilter package of the common library to set up a path filter based on the CLI options. After the scan, this filter is passed to Report.ExcludePaths (report package) to remove excluded files from vulnerabilities and dependency files.

Implementation plan

Update gemnasium gitlab-org/security-products/analyzers/gemnasium!217 (closed)
- Make its finder use DS_EXCLUDED_PATHS, and add unit tests
- Change existing image test for DS_EXCLUDED_PATHS, and introduce a broken file to prove that it's skipped (no parsing error)
Update gemnasium-python
- Update dependencies: gemnasium module
- Change main.go similar to what's been done in gemnasium project
- Add image test for DS_EXCLUDED_PATHS (using broken requirements.txt) if image tests are available; see #333829 (closed)
Update gemnasium-maven
- Port the changes to gemnasium-python.
- Also skip extra files returned by the builders, if they match DS_EXCLUDED_PATHS.

Permissions and Security

No change

Documentation

TBD. See discussion.

Availability & Testing

DS_EXCLUDED_PATHS is already tested in the context of gemnasium but there's no distinction b/w pre-filter and post-filtering, so we need an image test where a broken file is skipped, and the scan is successful.

This should be ported to gemnasium-maven and gemnasium-python (with broken dep. file or broken build file) once they have image integration tests. See #333829 (closed).

Available Tier

GitLab Ultimate

What does success look like, and how can we measure that?

Users no longer need a workaround to skip files that they don't want to scan, in particular in the context of gemnasium-maven and gemnasium-python since these only scan a single project.