poetry.lock support for Dependency Scanning

Problem to solve

Add Python Poetry support to Dependency Scanning, parse and process Poetry lock files.

See #32774 for supporting Poetry projects having no lock file.

Right now gemnasium-python fails because it detects the Poetry project, but then tries to install its dependencies instead of parsing poetry.lock, and it can't find a compatible builder.

[INFO] [gemnasium-python] [2022-04-14T13:19:01Z] ▶ GitLab gemnasium-python analyzer v2.23.0
[INFO] [gemnasium-python] [2022-04-14T13:19:01Z] ▶ Detected supported dependency files in '.'. Dependency files detected in this directory will be processed. Dependency files in other directories will be skipped.
[ERRO] [gemnasium-python] [2022-04-14T13:19:01Z] ▶ No builder for package manager poetry
[FATA] [gemnasium-python] [2022-04-14T13:19:01Z] ▶ no builder for requirements file

Further details

Poetry is a package manager and builder for Python, which is arguably better than pure pip. It generates poetry.lock, a lock file that gives the full, accurate list of the package versions a Python project depends on. Dependency Scanning should leverage this lock file to detect and report vulnerable dependencies.

Proposal

• add poetry.lock parser to gemnasium; see gitlab-org/security-products/analyzers/gemnasium!66 (merged)
• create python-poetry test project
• add Poetry as a package manager detected by gemnasium's finder, and enable it in PresetGemnasiumPython.
• (see implementation plan) integrate the poetry.lock parser into gemnasium-python, and leverage the python-poetry project for QA
• (see implementation plan) update Dependency Scanning documentation

Implementation plan

• update gemnasium-python's main.go to skip the "build" step when there's no builder, but a lock file parser is available. Right now it fails in that case.
• pull poetry.lock parser into gemnasium-python similar to pipdeptree: https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-python/-/blob/master/main.go#L30
• update image spec https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-python/-/blob/master/spec/image_spec.rb to add test for poetry (can use lock file from https://gitlab.com/gitlab-org/security-products/tests/python-poetry/-/blob/master/poetry.lock)
• update CI template to trigger gemnasium-python-dependency_scanning when there's a poetry.lock, and test this in dependency_scanning_gitlab_ci_yaml_spec.rb
• update documentation
  • document support for poetry: https://docs.gitlab.com/ee/user/application_security/dependency_scanning/#supported-languages-and-package-managers
  • document version tested: https://docs.gitlab.com/ee/user/application_security/dependency_scanning/#obtaining-dependency-information-by-parsing-lockfiles
  • remove additional support section: https://docs.gitlab.com/ee/user/application_security/dependency_scanning/#support-for-additional-languages
  • remove workaround for poetry: https://docs.gitlab.com/ee/user/application_security/dependency_scanning/#working-around-missing-support-for-certain-languages-or-package-managers

Availability & testing

python-poetry test project should be updated to test poetry.lock, test should pass.

What does success look like, and how can we measure that?

Dependency Scanning CI job successfully process Poetry projects having a lock file instead of failing with "unsupported language or manager" error.

Links / references

• Poetry