Refactor using builder package
What does this MR do?
Introduce builder
packages for pip, pipenv, and setuptools.
The analyze
function no longer has knowledge on the package managers the analyzer supports, except for the registration of the builders. It now proceeds in 5 steps:
- configure version range extensions, scanner, and builders
- look for supported project files, and stop after finding one
- delegate to the builder, in order to get a dependency list that can be parsed
- scan the this dependency list, and report vulnerabilities it as if they were coming from the project file
The function responsible for finding the compatible input files proceeds in 2 steps:
- it builds a map of all the filenames found in the target directory
- it iterates the patterns (combining builders and supported input files), and returns the first match
When finding compatible files, a builder wins over the ones that have been registered later. Also, the first input file returned by a builder wins over the others.
What are the relevant issue numbers?
gitlab-org/gitlab#263441 (closed)
Does this MR meet the acceptance criteria?
-
Changelog entry addedno behavior change Documentation created/updated, if necessary-
Tests added for this feature/bugcovered by QA Job definition updated, if necessary-
Conforms to the code review guidelines -
Conforms to the Go guidelines -
Security reports checked/validated by reviewer
What are the next steps?
- Detect lock files that can be parsed right away. Builders are skipped when supported lock files are detected. This makes it possible to add support for
Poetry.lock
andPipfile.lock
. See gitlab-org/gitlab#7006 (closed) and gitlab-org/gitlab#11756 (closed). - Introduce "freeze" plugins, and build lock files when they're missing, using some kind of
freeze
commands. Compatibility with freeze plugins is checked before iterating the builder plugins. This makes it possible to support forproject.toml
(Poetry). See gitlab-org/gitlab#32774 - Skip the build when
pipdeptree.json
is found. This makes it possible to run pipdeptree in a build job, and pass it on to the dependency scanning as a job artifact. - Scan multiple dependency files, found in different directory. Dependencies need to be installed in isolation using
virtualenv
or equivalent. See gitlab-org/gitlab#241659 - Enable/disable builders and other extensions at run-time. Each builder needs to have a name, so that it can be enabled or disabled using CLI flags. See gitlab-org/gitlab#198361 (closed)
Edited by Fabien Catteau