installing private packages using pip fails with 404

💼 Summary

1. Tried different scenarios and most of them are working.
2. A specific scenario (2 index urls with 2 different credentials) is failing due to what seems to be a bug in $ pip.
   • A workaround might be possible depending on the situation (using the same group deploy token for both index urls).
   • There is a related issue in the pip bug tracker.
   • From what I see (here and here) in the related fix MR, the situation will stay as-is for now on $ pip side.
3. The GitLab PyPI package registry documentation is incomplete:
   • Document that users can pull packages with deploy tokens and ci job tokens.
   • Document the bug happening in (2.).

☂ Conditions

I'm trying to reproduce this bug locally. My conditions are the following:

• pip version 22.3
• I have two packages (pypipackage1 and pypipackage2) in two different projects.

Between each scenario, I remove the packages and purge the cache with:

$ pip uninstall pypipackage1 pypipackage2 -y 
$ pip cache purge

Lastly, I need to use the --trusted-host option since my local server is not behind https.

Scenario index-url + extra-index-url with 2 different project tokens and their usernames

Note: the project deploy tokens have only the read_package_registry scope.

$ pip install pypipackage1 pypipackage2 --index-url http://<project_deploy_token_1_username>:<project_deploy_token_1>@gdk.test:8000/api/v4/projects/323/packages/pypi/simple --extra-index-url http://<project_deploy_token_2_username>:<project_deploy_token_2>@gdk.test:8000/api/v4/projects/324/packages/pypi/simple --trusted-host gdk.test

Looking in indexes: http://<project_deploy_token_1_username>:****@gdk.test:8000/api/v4/projects/323/packages/pypi/simple, http://<project_deploy_token_2_username>:****@gdk.test:8000/api/v4/projects/324/packages/pypi/simple
Collecting pypipackage1
  ERROR: HTTP error 404 while getting http://gdk.test:8000/api/v4/projects/323/packages/pypi/files/7c454e93e065f153c2a43640b853270ac183801a0dafb876d4874ff641bd4ecc/pypipackage1-1.3.8-py3-none-any.whl#sha256=7c454e93e065f153c2a43640b853270ac183801a0dafb876d4874ff641bd4ecc (from http://gdk.test:8000/api/v4/projects/323/packages/pypi/simple/pypipackage1/)
ERROR: Could not install requirement pypipackage1 from http://gdk.test:8000/api/v4/projects/323/packages/pypi/files/7c454e93e065f153c2a43640b853270ac183801a0dafb876d4874ff641bd4ecc/pypipackage1-1.3.8-py3-none-any.whl#sha256=7c454e93e065f153c2a43640b853270ac183801a0dafb876d4874ff641bd4ecc because of HTTP error 404 Client Error: Not Found for url: http://gdk.test:8000/api/v4/projects/323/packages/pypi/files/7c454e93e065f153c2a43640b853270ac183801a0dafb876d4874ff641bd4ecc/pypipackage1-1.3.8-py3-none-any.whl for URL http://gdk.test:8000/api/v4/projects/323/packages/pypi/files/7c454e93e065f153c2a43640b853270ac183801a0dafb876d4874ff641bd4ecc/pypipackage1-1.3.8-py3-none-any.whl#sha256=7c454e93e065f153c2a43640b853270ac183801a0dafb876d4874ff641bd4ecc (from http://gdk.test:8000/api/v4/projects/323/packages/pypi/simple/pypipackage1/)

This fails 💥

Looking closely (eg. with a proxy) at the request made for http://gdk.test:8000/api/v4/projects/323/packages/pypi/files/7c454e93e065f153c2a43640b853270ac183801a0dafb876d4874ff641bd4ecc/pypipackage1-1.3.8-py3-none-any.whl, I saw that the Authorization is filled with the value of <project_deploy_token_2>. This will fail because that request is for pypipackage1 that lives in project 1 = <project_deploy_token_2> can't read it.

$ pip seems to go like this:

• For pypipackage1:
  1. Check the index-url with the credentials provided (deploy token 1). That's a 200 OK.
  2. Check the extra-index-url with the credentials provided (deploy token 2). Nevermind the 302, that's the request forward feature in action.
  3. Take the response from (1.). Check for the .whl url.
  4. Contact the .whl url with the credentials of (2.)... 💥 (404 not found) as deploy token 2 doesn't work on project 1.

This seems to be a bug in $ pip. If I need to toss a guess, it would:

• $ pip reads the arguments --index-url, --extra-index-url and extract the credentials (the deploy tokens).
• $ pip will then store those credentials using the host + port as a key, in this case gdk.test:8000. Since, both arguments are pointing to the same host, the last argument credentials will be stored in the same place as the first arguments:
  • The above means that the deploy token 2 in the command line option is effectively overwriting the deploy token 1.
• Then when $ pip needs to contact gdk.test:8000, it will look for the credentials store for that key and 💥 that's deploy token 2.

🚒 Possible workaround

So the problem is when several projects (located in the same GitLab instance) are targeted with multiple index urls on $ pip and different credentials are used.

One possible workaround is: if those projects share a common group ancestor then it is possible to create a group deploy token there (the minimal scope is read_package_registry) and use that token for all index urls.

🔮 Conclusions

From my quick trials above, I see the following:

1. When using multiple index urls (--index-url + --extra-index-url) with same host and different credentials, $ pip seems to mix up credentials and fails to download the files because it uses the wrong credential during that file download.
   • This has to be fixed on $ pip side: it is sending the wrong credentials.
2. The GitLab documentation our pulling packages is lacking details:
   • deploy tokens and ci job tokens are supported but yet not documented for pulling.
   • we should document this bug happening with $ pip in (1.).

❓ Questions

• Why has this 💥 only a few months ago.
  • In %15.4, we deployed a security fix for the PyPI Repository: on file download endpoints, we enforced the credentials (see https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/2592#key-pypi-authentication (internal)). Since in scenario 5️⃣, $ pip is sending the wrong ones, yeah GitLab will now complain and 💥 (404 Not Found).

Tested and working scenarios

1️⃣ Scenario index-url + extra-index-url with personal access token and

$ pip install pypipackage1 pypipackage2 --index-url http://<username>:<personal_access_token>@gdk.test:8000/api/v4/projects/323/packages/pypi/simple --extra-index-url http://<username>:<personal_access_token>@gdk.test:8000/api/v4/projects/324/packages/pypi/simple --trusted-host gdk.test
Looking in indexes: http://<username>:****@gdk.test:8000/api/v4/projects/323/packages/pypi/simple, http://<username>:****@gdk.test:8000/api/v4/projects/324/packages/pypi/simple
Collecting pypipackage1
  Downloading http://gdk.test:8000/api/v4/projects/323/packages/pypi/files/7c454e93e065f153c2a43640b853270ac183801a0dafb876d4874ff641bd4ecc/pypipackage1-1.3.8-py3-none-any.whl (1.4 kB)
Collecting pypipackage2
  Downloading http://gdk.test:8000/api/v4/projects/324/packages/pypi/files/a896b5cb500a881e248b0d05fbab42ca458d99a5fa5393a60a28779c2b4d8d7e/pypipackage2-2.5.6-py3-none-any.whl (1.4 kB)
Installing collected packages: pypipackage2, pypipackage1
Successfully installed pypipackage1-1.3.8 pypipackage2-2.5.6

It works ✅

2️⃣ Scenario index-url + extra-index-url with personal access token and __token__

$ pip install pypipackage1 pypipackage2 --index-url http://__token__:<personal_access_token>@gdk.test:8000/api/v4/projects/323/packages/pypi/simple --extra-index-url http://__token__:<personal_access_token>@gdk.test:8000/api/v4/projects/324/packages/pypi/simple --trusted-host gdk.test
Looking in indexes: http://__token__:****@gdk.test:8000/api/v4/projects/323/packages/pypi/simple, http://__token__:****@gdk.test:8000/api/v4/projects/324/packages/pypi/simple
Collecting pypipackage1
  Downloading http://gdk.test:8000/api/v4/projects/323/packages/pypi/files/7c454e93e065f153c2a43640b853270ac183801a0dafb876d4874ff641bd4ecc/pypipackage1-1.3.8-py3-none-any.whl (1.4 kB)
Collecting pypipackage2
  Downloading http://gdk.test:8000/api/v4/projects/324/packages/pypi/files/a896b5cb500a881e248b0d05fbab42ca458d99a5fa5393a60a28779c2b4d8d7e/pypipackage2-2.5.6-py3-none-any.whl (1.4 kB)
Installing collected packages: pypipackage2, pypipackage1
Successfully installed pypipackage1-1.3.8 pypipackage2-2.5.6

It works ✅

3️⃣ Scenario extra-index-url with personal access token and username

$ pip install pypipackage2 --extra-index-url http://<username>:<personal_access_token>@gdk.test:8000/api/v4/projects/324/packages/pypi/simple --trusted-host gdk.test        
Looking in indexes: https://pypi.org/simple, http://<username>:****@gdk.test:8000/api/v4/projects/324/packages/pypi/simple
Collecting pypipackage2
  Downloading http://gdk.test:8000/api/v4/projects/324/packages/pypi/files/a896b5cb500a881e248b0d05fbab42ca458d99a5fa5393a60a28779c2b4d8d7e/pypipackage2-2.5.6-py3-none-any.whl (1.4 kB)
Installing collected packages: pypipackage2
Successfully installed pypipackage2-2.5.6

It works ✅

4️⃣ Scenario extra-index-url with personal access token and __token__

$ pip install pypipackage2 --extra-index-url http://__token__:<personal_access_token>@gdk.test:8000/api/v4/projects/324/packages/pypi/simple --trusted-host gdk.test
Looking in indexes: https://pypi.org/simple, http://__token__:****@gdk.test:8000/api/v4/projects/324/packages/pypi/simple
Collecting pypipackage2
  Downloading http://gdk.test:8000/api/v4/projects/324/packages/pypi/files/a896b5cb500a881e248b0d05fbab42ca458d99a5fa5393a60a28779c2b4d8d7e/pypipackage2-2.5.6-py3-none-any.whl (1.4 kB)
Installing collected packages: pypipackage2
Successfully installed pypipackage2-2.5.6

It works ✅