Support multiple dependency config files for Python Pip
What does this MR do and why?
Previous MRs
In !162313 (merged), we introduced the ConfigFiles::Base
class where the intention is for each dependency manager config file type to be represented by a child class. The child class contains the parsing logic to extract a list of libraries and their versions from the file
In !162891 (merged), we added the ConfigFileParser
class which searches through the project repository for dependency config files, matches them to the applicable config file class, and then parses them.
Context
Currently, we only process one file per config file class. However, in #491800 (closed), we found that the Python Pip dependency manager often has "nested" requirements files. For example, its primary config file requirements.txt
may include another file with -r other_requirements.txt
.
Instead of directly supporting nested files, we decided to expand the functionality of ConfigFileParser
to support multiple files per config file class. This approach relies on the convention that all Pip-related requirements files have the word requirements
in the name.
This MR
This MR is the first step in expanding the functionality of our config file parser. It includes these changes:
- Introduces a new class method to
ConfigFiles::Base
that indicates if it should support multiple files. - Updates
ConfigFileParser
so that it will find all matching files in the repo for config file classes that support them.
In the next MR, we will update Ai::RepositoryXray::ScanDependenciesService
so that it will combine the payloads of config files belonging to the same language. (Only one X-ray report per language can be saved, so that's why we'll merge the payloads.) It will also include the documentation update for Python Pip.
Resolves part of #491800 (closed)
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
How to set up and validate locally
- Create a new blank project and add the following files to the repo:
requirements.txt
:
pytest >= 2.6.4 ; python_version < '3.8'
openpyxl == 3.1.2
-i https://pypi.org/simple
--python-version 3
subdir/dev-requirements.txt
:
python_dateutil>=2.5.3
fastapi-health!=0.3.0
- Run the config file parser directly in the Rails console:
project = Project.last # Should be the project you created in Step 1
result = Ai::Context::Dependencies::ConfigFileParser.new(project).extract_config_files
result.map { |r| { class: r.class.name, payload: r.payload } }
- Observe that it outputs two objects for the same config file class.
Related to #491800 (closed)