Skip to content

Add PyPI package requests forward

David Fernandez requested to merge 233413-pypi-forwarding into master

🎟 Context

The npm package registry has a feature called the package request forward. It can be summarized as:

When a package is requested to the GitLab package registry and the package doesn't exist in the accessed resource, forward that request to the official registry.

This behavior is controlled with an application setting which is enabled by default on gitlab.com.

PyPI users voiced their interest on this feature. That's issue #233413 (closed).

🔍 What does this MR do?

This MR is an exact copy of the npm package request forward for pypi packages.

  • Update the pypi finder so that it can't raise an ActiveRecord::NotFound error.
  • Update the metadata API endpoints so that if the given package name is not found, the request gets redirected.
  • Update the dependency proxy helpers to support different package types.
  • Add an application setting that gates this behavior.
  • Update/add all related specs.

A small word on the EE side. For some reason, this application setting is marked as premium. See https://docs.gitlab.com/ee/user/admin_area/settings/continuous_integration.html#npm-forwarding. The issue is that package features have been moved to Core. As such, the API endpoints when using the package forward function, they don't check the license features.

In other words, package request forward is in the EE side but it should be moved to Core. This is way outside of this MR and we opened #337862 for that.

For consistency (with the existing similar npm feature), the changes here are in EE but keep in mind that we're going to remove those very soon (currently scheduled for %14.4)

📸 Screenshots or Screencasts (strongly suggested)

Assume that we have a project with a single pypi package:

Project.find(38).packages.pypi.map(&:name)
=> ["pypibananas"]

Project level

Let's try to pull it using the project level instance:

$ pip install --trusted-host gdk.test --index-url http://root:XXXX@gdk.test:8000/api/v4/projects/38/packages/pypi/simple --no-deps pypibananas

DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
Looking in indexes: http://root:****@gdk.test:8000/api/v4/projects/38/packages/pypi/simple
Collecting pypibananas
  Downloading http://gdk.test:8000/api/v4/projects/38/packages/pypi/files/26daf30525a79d6c75dd93e16f469acc050ed315d90434b09e218345ba671b3c/pypibananas-1.3.7-py3-none-any.whl (1.4 kB)
Installing collected packages: pypibananas
DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
Successfully installed pypibananas-1.3.7

It works ✅

Now with the application setting enabled, let's try to pull a package that is not in the project:

$ pip install --trusted-host gdk.test --index-url http://root:XXXX@gdk.test:8000/api/v4/projects/38/packages/pypi/simple --no-deps pytmi      

DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
Looking in indexes: http://root:****@gdk.test:8000/api/v4/projects/38/packages/pypi/simple
Collecting pytmi
  Using cached pytmi-0.2.0-py3-none-any.whl (6.8 kB)
Installing collected packages: pytmi
DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
Successfully installed pytmi-0.2.0

It works ✅

Let's now disable the application setting and try again:

$ pip uninstall pytmi
$ pip install --trusted-host gdk.test --index-url http://root:XXXX@gdk.test:8000/api/v4/projects/38/packages/pypi/simple --no-deps pytmi

DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
Looking in indexes: http://root:****@gdk.test:8000/api/v4/projects/38/packages/pypi/simple
ERROR: Could not find a version that satisfies the requirement pytmi (from versions: none)
ERROR: No matching distribution found for pytmi

It works ✅ . The request is not forwarded and thus a not found is returned by the backend.

Lastly, with the application enabled, let's try to pull a package that doesn't exist anywhere (project or the official registry):

$ pip install --trusted-host gdk.test --index-url http://root:XXXX@gdk.test:8000/api/v4/projects/38/packages/pypi/simple --no-deps idonotexist

DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
Looking in indexes: http://root:****@gdk.test:8000/api/v4/projects/38/packages/pypi/simple
ERROR: Could not find a version that satisfies the requirement idonotexist (from versions: none)
ERROR: No matching distribution found for idonotexist

Package not found ✅

Group level

Let's run the same cases but with the group level endpoint.

Pulling the existing package:

$ pip uninstall pypibananas
$ pip install --trusted-host gdk.test --index-url http://root:XXXX@gdk.test:8000/api/v4/groups/113/-/packages/pypi/simple --no-deps pypibananas

DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
Looking in indexes: http://root:****@gdk.test:8000/api/v4/groups/113/-/packages/pypi/simple
Collecting pypibananas
  Downloading http://gdk.test:8000/api/v4/groups/113/-/packages/pypi/files/b6c1510e1289a906bfaa5b551ea95ae6570f87bc0b18ab753de6bf38b9cbc1c7/pypibananas-1.3.7-py3-none-any.whl (1.4 kB)
Installing collected packages: pypibananas
DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
Successfully installed pypibananas-1.3.7

It works ✅

Now with the application setting enabled, let's try to pull a package that is not in the group:

$ pip uninstall pytmi
$ pip install --trusted-host gdk.test --index-url http://root:XXXX@gdk.test:8000/api/v4/groups/113/-/packages/pypi/simple --no-deps pytmi      

DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
Looking in indexes: http://root:****@gdk.test:8000/api/v4/groups/113/-/packages/pypi/simple
Collecting pytmi
  Using cached pytmi-0.2.0-py3-none-any.whl (6.8 kB)
Installing collected packages: pytmi
DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
Successfully installed pytmi-0.2.0

Works ✅

Let's now disable the application setting and try again:

$ pip uninstall pytmi
$ pip install --trusted-host gdk.test --index-url http://root:XXXX@gdk.test:8000/api/v4/groups/113/-/packages/pypi/simple --no-deps pytmi

DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
Looking in indexes: http://root:****@gdk.test:8000/api/v4/groups/113/-/packages/pypi/simple
ERROR: Could not find a version that satisfies the requirement pytmi (from versions: none)
ERROR: No matching distribution found for pytmi

Package not found ✅

Lastly, with the application enabled, let's try to pull a package that doesn't exist anywhere (project or the official registry):

$ pip install --trusted-host gdk.test --index-url http://root:XXXX@gdk.test:8000/api/v4/groups/113/-/packages/pypi/simple --no-deps idonotexist

DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
Looking in indexes: http://root:****@gdk.test:8000/api/v4/groups/113/-/packages/pypi/simple
ERROR: Could not find a version that satisfies the requirement idonotexist (from versions: none)
ERROR: No matching distribution found for idonotexist

Package not found ✅

⚗ How to setup and validate locally (strongly suggested)

  • Have a personal access token with the api scope ready.
  • Enable or disable the application setting in: /admin/application_settings/ci_cd#js-package-settings.
  • Install pip and try the commands above.

📐 Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

Does this MR contain changes to processing or storing of credentials or tokens, authorization and authentication methods or other items described in the security review guidelines? If not, then delete this Security section.

  • [-] Label as security and @ mention @gitlab-com/gl-security/appsec
  • [-] The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • [-] Security reports checked/validated by a reviewer from the AppSec team

💽 Database review

Up

$ rails db:migrate
== 20210806152104 AddPypiPackageRequestsForwardingToApplicationSettings: migrating 
-- add_column(:application_settings, :pypi_package_requests_forwarding, :boolean, {:default=>true, :null=>false})
   -> 0.0041s
== 20210806152104 AddPypiPackageRequestsForwardingToApplicationSettings: migrated (0.0120s)

Down

$ rails db:rollback
== 20210806152104 AddPypiPackageRequestsForwardingToApplicationSettings: reverting 
-- remove_column(:application_settings, :pypi_package_requests_forwarding)
   -> 0.0059s
== 20210806152104 AddPypiPackageRequestsForwardingToApplicationSettings: reverted (0.0182s)
Edited by David Fernandez

Merge request reports