Skip to content

Support multiple dependency config files for Python Pip

What does this MR do and why?

Previous MRs

In !162313 (merged), we introduced the ConfigFiles::Base class where the intention is for each dependency manager config file type to be represented by a child class. The child class contains the parsing logic to extract a list of libraries and their versions from the file

In !162891 (merged), we added the ConfigFileParser class which searches through the project repository for dependency config files, matches them to the applicable config file class, and then parses them.

Context

Currently, we only process one file per config file class. However, in #491800 (closed), we found that the Python Pip dependency manager often has "nested" requirements files. For example, its primary config file requirements.txt may include another file with -r other_requirements.txt.

Instead of directly supporting nested files, we decided to expand the functionality of ConfigFileParser to support multiple files per config file class. This approach relies on the convention that all Pip-related requirements files have the word requirements in the name.

This MR

This MR is the first step in expanding the functionality of our config file parser. It includes these changes:

  • Introduces a new class method to ConfigFiles::Base that indicates if it should support multiple files.
  • Updates ConfigFileParser so that it will find all matching files in the repo for config file classes that support them.

In the next MR, we will update Ai::RepositoryXray::ScanDependenciesService so that it will combine the payloads of config files belonging to the same language. (Only one X-ray report per language can be saved, so that's why we'll merge the payloads.) It will also include the documentation update for Python Pip.

Resolves part of #491800 (closed)

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

How to set up and validate locally

  1. Create a new blank project and add the following files to the repo:

requirements.txt:

pytest >= 2.6.4 ; python_version < '3.8'
openpyxl == 3.1.2

-i https://pypi.org/simple
--python-version 3

subdir/dev-requirements.txt:

python_dateutil>=2.5.3
fastapi-health!=0.3.0
  1. Run the config file parser directly in the Rails console:
project = Project.last # Should be the project you created in Step 1

result = Ai::Context::Dependencies::ConfigFileParser.new(project).extract_config_files
result.map { |r| { class: r.class.name, payload: r.payload } }
  1. Observe that it outputs two objects for the same config file class.

Screenshot_2024-09-24_at_6.58.31_PM

Related to #491800 (closed)

Edited by Leaminn Ma

Merge request reports

Loading