Add Dependency Scanning support for Python 3.12

Context

Python 3.12 has been released as production ready.

The resolution of packages is highly dependent on the version of Python used, so it's important to provide images that are equivalent to the versions running in production for customers.

Not doing so can lead to bugs or inaccurate dependency resolution that results in false negatives and false positives once advisory matching takes place.

Links

• Python version status
  • 3.12 has been released outside of pre-release stages.

Proposal

Our current approach to support additional versions of python is to generate a different container image that can be used as a replacement for the Dependency Scanning CI job. Though, this approach incurs additional maintenance costs for our development team. Alternatives have been discussed and the following solutions have been suggested:

1. supporting pip-compile lock files Handle requirements.txt files produced by pip-c... (#418321 - closed):
   • it would make the scan a lot faster and skip the dependencies installation.
   • It's more accurate. The scanner knows the exact versions used by the project.
   • It's compatible with any version of Python, and removes potential compatibility issues.
   • It works with offline environments out of the box, because the analyzer doesn't download any packages.
2. supporting CycloneDX SBOM generators other than Gemnasium Spike: Replace Gemnasium with open source nativ... (#434143).
   • Users can easily integrate the tool they want to generate an SBOM report
   • The SCA features don't need to know anything about the project's build specificities, and thus any complexity in supporting them is avoided.

Implementation

TODO

added Category:Software Composition Analysis GitLab Ultimate documentation featureaddition groupcomposition analysis sectionsec typefeature workflowrefinement labels

added devopssecure label

changed the description

@hacks4oats would it be more efficient for the team if we add support for both 3.11 and 3.12 in the same issue, or would you recommend handling those separately? Could you please add a high-level estimate for how much work this is?

changed milestone to %Backlog

Reporting here a feedback from the customer in an Ultimate trial that is currently experiencing issues with the scanners not supporting 3.11:

It looks like gemnasium tries to install the content of the requirements.txt file to generate the complete list of dependencies. Which makes it specific to a particular python or even pip version. In our case, as we declare dependencies in a requirements.in file and generate the requirements.txt with hashes, requirements.txt necessarily contains all the dependencies.

If gemnasium could detect this format, it could directly verify the versions without having to install them like other package managers do (e.g npm for packag.json). Thus the tool would be compatible with any version of python or pip and it would be easier to generate multiple requirements files like requirements.txt

@smeadzinger do you think this should be reported in a separate issue or could that help with the support of this issue? I am happy to ask for more info from the customer or put you in touch if needed.

GitLab Ultimate customer is interested in seeing this implemented.

• ZD (internal use)
• SF (internal use)

added customer label

Python 3.11 support in Gemnasium-python been requested by a GitLab Ultimate customer on confidential ticket 480994. FYI @mmatayoshi

mentioned in merge request gitlab-org/security-products/analyzers/gemnasium!620 (closed)

mentioned in merge request !141100 (closed)

The following customer is interested in this capability

• Subscription: GitLab Ultimate
• Product: gitlab.com
• Link to request: https://gitlab.zendesk.com/agent/tickets/502144 (internal)
• Why interested: Projects are using Python 3.11

With Ubuntu 24.04 being released soon I can see this becoming a bigger issue. As more people will start to run into this if the SCA scanner doesn't support the default version of Python for the latest Ubuntu LTS version.

The following customer is interested in this capability

• Subscription: GitLab Premium
• Product: gitlab.com
• Link to request: https://gitlab.zendesk.com/agent/tickets/505513 (internal)
• Why interested: The new Ubuntu LTS version will be using python 3.11

marked this issue as related to #440183

changed title from Add support for Python 3.11 to gemnasium-python to Add support for Python 3.11 (and 3.12) to gemnasium-python

changed the description

mentioned in issue #441491 (closed)

Hi, as Gitlab Ultimate subscriber we are also interested in having this working. Dependency vulnerability management is one of the main reasons why we upgraded to this tier, and our expectation was that Python10.11 will be supported

changed title from Add support for Python 3.11 (and 3.12) to gemnasium-python to Add support for Python 3.12 to gemnasium-python

marked this issue as related to #441491 (closed)

changed the description

changed title from Add support for Python 3.12 to gemnasium-python to Add Dependency Scanning support for Python 3.12

@johncrowley, based on @zmartins' tests of various tools to generate Python CDX SBOM, the results are on part with our Gemnasium tool.

What do you think of using this issue to create an experimental CI/CD Component for Python DS using a 3rd party tool?

We could direct interested users from this thread to try it. The feedback should be relevant to the discussion in Discuss new UX for enabling Dependency Scanning... (#458920 - closed).

/cc @fcatteau @gonzoyumo

The following customer is interested in this capability

• Subscription: GitLab Ultimate
• Product: self-managed
• Link to request: https://gitlab.zendesk.com/agent/tickets/569953 (internal)

The customer is asking if there are any updates or timelines on support for Python 3.12.

cc @thiagocsf

Thanks for the tag, @bnyaringita

I've closed this because the new dependency scanning experience available from 17.5 will not depend on the Python version.

The zendesk ticket doesn't specify details about the customer's python project. Right now, we support Pipfile.lock and pipenv.graph.json (both generated by pipenv), poetry.lock (generated by poetry), and requirements.txt (generated by pip). The full list is currently listed in https://gitlab.com/gitlab-org/security-products/analyzers/dependency-scanning#supported-files while we work on the user docs.

Additional support for python is being tracked in Report vulnerable dependency paths for pip, set... (#229816 - closed)

If the customer is interested in trying the new analyzer, please let me know. I left a comment here with basic instructions: &12313 (comment 2122781881)

Due to &14146, and &1463, this issue is no longer relevant.

closed

added closedwill not do label