Make Gemnasium skip files listed in DS_EXCLUDED_PATHS
Release notes
Variable DS_EXCLUDED_PATHS behaviour changed to pre-filter
For users of the Depedency Scanning variable DS_EXCLUDED_PATHS we are introducing a breaking change, it will now pre-filter. This change applied to all versions asthe change is being backported. December 22, 2021 forward gemnasium considers DS_EXCLUDED_PATHS when searching for supported projects and will pre-filter out those that match. Pre-filtering prevents the analyzer from logging warning or failing when processing dependency files that have been explicitly excluded using DS_EXCLUDED_PATHS.
You should not need to take any action, however if you were expecting this post-filtering behaviour you will no longer have this data available to you in logs or artifacts.
The previous behaviour was causing failures and unexpected errors for users and after discussions we found that this, pre-filter as opposed to post filter, was the more expected and desired behaviour.
Problem to solve
Gemnasium can be configure to filter specific files and directories using the DS_EXCLUDED_PATHS
variable, but this is implemented as a post-filter, which causes issues:
- when the excluded build file, dependency file, or lock file can't be processed
- when the analyzer only scans one file/directory, and it automatically picks up the one being excluded
The workaround is to remove the files being excluded prior to the scan, in the before_script
of the scanning job.
Intended users
User experience goal
Proposal
Make Gemnasium consider DS_EXCLUDED_PATHS
when searching for supported projects, and skip files that match this CI file (pre-filtering). Pre-filtering would prevent the analyzer from logging warning or even failing when processing dependency files that have been explicitly excluded using DS_EXCLUDED_PATHS
.
Further details
Gemnasium relies on the pathfilter
package of the common library to set up a path filter based on the CLI options. After the scan, this filter is passed to Report.ExcludePaths
(report
package) to remove excluded files from vulnerabilities and dependency files.
Implementation plan
-
Update gemnasium gitlab-org/security-products/analyzers/gemnasium!217 (closed) - Make its
finder
useDS_EXCLUDED_PATHS
, and add unit tests - Change existing image test for
DS_EXCLUDED_PATHS
, and introduce a broken file to prove that it's skipped (no parsing error)
- Make its
-
Update gemnasium-python - Update dependencies:
gemnasium
module - Change
main.go
similar to what's been done ingemnasium
project - Add image test for
DS_EXCLUDED_PATHS
(using brokenrequirements.txt
) if image tests are available; see #333829 (closed)
- Update dependencies:
-
Update gemnasium-maven - Port the changes to
gemnasium-python
. - Also skip extra files returned by the builders, if they match
DS_EXCLUDED_PATHS
.
- Port the changes to
Permissions and Security
No change
Documentation
TBD. See discussion.
Availability & Testing
DS_EXCLUDED_PATHS
is already tested in the context of gemnasium
but there's no distinction b/w pre-filter and post-filtering, so we need an image test where a broken file is skipped, and the scan is successful.
This should be ported to gemnasium-maven and gemnasium-python (with broken dep. file or broken build file) once they have image integration tests. See #333829 (closed).
Available Tier
What does success look like, and how can we measure that?
Users no longer need a workaround to skip files that they don't want to scan, in particular in the context of gemnasium-maven and gemnasium-python since these only scan a single project.
What is the type of buyer?
Is this a cross-stage feature?
No