Skip to content

Dependency scanning fails for setuptools project that includes pyproject.toml

Summary

Python projects that use setuptools for metadata (i.e. have a setup.cfg or setup.py file) but also include a pyproject.toml file cannot be scanned properly by the dependency scanning template.

Note that including a pyproject.toml file to declare the build-system is the recommended way to configure a project for setuptools.

Steps to reproduce

Configure a project with the following files:

pyproject.toml 
[build-system]
requires = [
    "setuptools>=42",
    "wheel",
]
build-backend = "setuptools.build_meta"
setup.py 
from setuptools import setup
setup(
    name="testproject",
    version="0.0.1",
    install_requires=["coloredlogs"],
)
.gitlab-ci.yml 
include:
  # https://docs.gitlab.com/ee/user/application_security/dependency_scanning/
  - template: Jobs/Dependency-Scanning.gitlab-ci.yml

dependency_scanning:
  variables:
    SECURE_LOG_LEVEL: "debug"

Example Project

https://gitlab.com/duncanmmacleod/dependency-scanning-test/-/tree/setup.py

What is the current bug behavior?

gemnasium-python see pyproject.toml and presumes the project is a poetry project, and ignores the setup.py file.

What is the expected correct behavior?

Since pyproject.toml is independent of the build infrastructure it should be selected after setup.{cfg,py}, probably last except for requirements.txt.

Or, the scanner should read the pyproject.toml file and use the [build-system]/build-backend option to determine the package manager.

Relevant logs and/or screenshots

From https://gitlab.com/duncanmmacleod/dependency-scanning-test/-/jobs/2325799205:

$ /analyzer run
[INFO] [gemnasium-python] [2022-04-12T12:57:35Z] ▶ GitLab gemnasium-python analyzer v2.22.0
[DEBU] [gemnasium-python] [2022-04-12T12:57:35Z] ▶ inspect directory: .
[DEBU] [gemnasium-python] [2022-04-12T12:57:35Z] ▶ skip ignored directory: .git
[DEBU] [gemnasium-python] [2022-04-12T12:57:35Z] ▶ electing poetry for pypi because this is the first match
[DEBU] [gemnasium-python] [2022-04-12T12:57:35Z] ▶ rejecting setup.py as handled by setuptools
[DEBU] [gemnasium-python] [2022-04-12T12:57:35Z] ▶ electing poetry for pypi because this is the first match
[DEBU] [gemnasium-python] [2022-04-12T12:57:35Z] ▶ rejecting setup.py as handled by setuptools
[INFO] [gemnasium-python] [2022-04-12T12:57:35Z] ▶ Detected supported dependency files in '.'. Dependency files detected in this directory will be processed. Dependency files in other directories will be skipped.
[DEBU] [gemnasium-python] [2022-04-12T12:57:35Z] ▶ Exporting dependencies for /builds/duncanmmacleod/dependency-scanning-test/pyproject.toml
[ERRO] [gemnasium-python] [2022-04-12T12:57:35Z] ▶ No builder for package manager poetry
[FATA] [gemnasium-python] [2022-04-12T12:57:35Z] ▶ no builder for requirements file

Output of checks

This bug happens on GitLab.com

Possible fixes

pyproject.toml is no longer specific to Poetry, so it should be removed from the definition of PackageManagerPoetry.

	// PackageManagerPoetry describes Poetry (Python)
	PackageManagerPoetry = PackageManager{
		Name:        "poetry",
		PackageType: PackageTypePypi,
		Files: []File{
			{Filename: "pyproject.toml", FileType: FileTypeRequirements},
			{Filename: "poetry.lock", FileType: FileTypeLockFile},
		},
	}

See https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/blob/4785f7789e1ab41948b2c1ed786e6b9469fff12e/finder/package_manager.go#L196

Implementation plan

Update gemnasium-python:

  • Remove pyproject.toml from PackageManagerPoetry.
  • Add an integration test for the case where a setuptool project has setup.py and pyproject.toml.
  • Release a new version.
Edited by Fabien Catteau