Skip to content

Refactor using builder package

Fabien Catteau requested to merge 263441-builder-pkg into master

What does this MR do?

Introduce builder packages for pip, pipenv, and setuptools.

The analyze function no longer has knowledge on the package managers the analyzer supports, except for the registration of the builders. It now proceeds in 5 steps:

  1. configure version range extensions, scanner, and builders
  2. look for supported project files, and stop after finding one
  3. delegate to the builder, in order to get a dependency list that can be parsed
  4. scan the this dependency list, and report vulnerabilities it as if they were coming from the project file

The function responsible for finding the compatible input files proceeds in 2 steps:

  • it builds a map of all the filenames found in the target directory
  • it iterates the patterns (combining builders and supported input files), and returns the first match

When finding compatible files, a builder wins over the ones that have been registered later. Also, the first input file returned by a builder wins over the others.

What are the relevant issue numbers?

gitlab-org/gitlab#263441 (closed)

Does this MR meet the acceptance criteria?

What are the next steps?

  • Detect lock files that can be parsed right away. Builders are skipped when supported lock files are detected. This makes it possible to add support for Poetry.lock and Pipfile.lock. See gitlab-org/gitlab#7006 (closed) and gitlab-org/gitlab#11756 (closed).
  • Introduce "freeze" plugins, and build lock files when they're missing, using some kind of freeze commands. Compatibility with freeze plugins is checked before iterating the builder plugins. This makes it possible to support for project.toml (Poetry). See gitlab-org/gitlab#32774
  • Skip the build when pipdeptree.json is found. This makes it possible to run pipdeptree in a build job, and pass it on to the dependency scanning as a job artifact.
  • Scan multiple dependency files, found in different directory. Dependencies need to be installed in isolation using virtualenv or equivalent. See gitlab-org/gitlab#241659
  • Enable/disable builders and other extensions at run-time. Each builder needs to have a name, so that it can be enabled or disabled using CLI flags. See gitlab-org/gitlab#198361 (closed)
Edited by Fabien Catteau

Merge request reports