Dependency Proxy for packages: metadata endpoints

🔥 Problem

The dependency proxy for packages not only needs to deal with package files but also metadata responses.

🤔 What are metadata responses?

Package manager clients will usually rely on dedicated endpoints to know the state of the package registry. Usually, it's to know things like:

• what are the packages available.
• given a package, what are the versions available.
• where are the package files located.

Depending on the package format, these endpoints can work at different level: registry wide, package name level or package name + version level. Most of the time, it is up to the registry to generate the correct response for these endpoints.

Here is a summary of all GA formats:

Format	Metadata level	Metadata format	Notes
Maven	xml	package	Content managed by clients.
NPM	json	package	Content managed by registry.
NuGet	json	package + registry	Content managed by registry.
PyPI	html	package version + package + registry	Content managed by registry.
Generic	-	-	-

😱 Dependency proxyfying metadata responses

In the context of dependency proxies, these endpoints become challenging because:

• The response can contain urls or references to the registry itself. This is a problem for the dependency proxy as we want to return response in such a way that clients will always go through the GitLab instance for all interactions.
• The main problem is that the contents of metadata response are more or less cached by clients. They can even use these contents to build the lock file (this is the file that will exactly list the packages (name+version) of a given application).
  • Take a CI job that build an application for example here. It will read the lock file to pull all the actual files. If the lock file references the upstream, the dependency proxy is completely avoided which is not what we want.

(2.) might be hard to understand. Let's imagine an example for package format foobar. Let's say that the registry has to implement a metadata endpoint packages/<package_name>/<package_version> that need to return the url of the package archive. Let's say that the official registry is located at https://registy.foobar.org, accessing https://registy.foobar.org/packages/test/1.3.7 might return https://registy.foobar.org/files/test-1.3.7.zip. All good here.

Now, let's say that we have a GitLab dependency proxy for foobar and we point to https://registy.foobar.org. Accessing https://gitlab.com/api/v4/projects/123/dependency_proxy/packages/foobar/packages/test/1.3.7 in the GitLab dependency proxy should return an url. If we simply take the response from the upstream and return it, the clients will still access https://registy.foobar.org and that's not what we really want. Instead, the answer of that endpoint should be https://gitlab.com/api/v4/projects/123/dependency_proxy/packages/foobar/....

In simple words, we have a client that asks for a metadata response and we need to do:

1. Pull that response from upstream.
2. Modify it to replace all references to upstream by references to the dependency proxy.
3. Return it back to the client while uploading it to the dependency proxy (caching). This part seems hard but it's already handled by workhorse.

The entire question is thus: how can we implement (1.) and (2.) in an efficient manner. Keep in mind that metadata response can be very large documents. Again some package formats will have a metadata response at the registry level. This means that the document will describe every package that is available in the registry. You can imagine that for public registries, this response is pretty large.

🗜️ Restrictions

How the Dependency Proxy feature works brings us some restrictions:

• The big one: we can't use background jobs.
  • When the dependency proxy needs to do this, there is a client waiting for the file. We can't serve the file from the upstream and have a background job modify it because that means that the first request receives a version and subsequent requests (after the background job ends) get a different version.
• The modification to do on files should be mainly replacing urls but it could be as complex as rewriting structure (like rewriting json structures).
• From the GA formats above, the formats we need to deal with are: json and html.
  • Bonus points of we can deal with the xml format. Future package formats (such as rpm) have xml metadata responses.
• Metadata responses can be large. Examples: metadata for npm itself is 22+ MB. The registry level metadata tends to be very, very large because it describes the entire registry state (all packages with all available versions).
• This needs to be fast as we have a web request waiting for an answer.
• This file is not fetched in one step and returned to the client in an other step. We leverage the workhorse dependencyproxy logic, where the file is streamed from the upstream back to the client.
  • This is to say that we have ruby and golang at our disposal.

The good news: we are going to cache this result so that next time the same metadata is requested, we can serve the cache entry.