Skip to content

GitLab Next

GitLab
GitLab
Closed
Opened by Joshua Lambert@joshlambert
Report abuse New issue

Dependency proxy for container registries

Problem to solve

The GitLab Container Registry allows users to to build and deploy docker images for usage in CI/CD pipelines or deployments to Kubernetes. However, each time an image was built dependencies would have to be fetched from external sources and downloaded. This introduced risk, as pipelines are dependent on 3rd party sources for each dependency. It would also slow down build times, as the build process was dependent on each download completing.

Target audience

Proposal

We will build a caching dependency proxy for container registries into the Rails codebase, at the group level. The proxy will receive requests and return the upstream image. Each time a new image is downloaded, it will be added to the Dependency Proxy and be cached for fast, reliable future downloads. The more you use the proxy, the faster your future builds will run.

Availability

This is the MVC and so there are availability limitations. Please see the below list:

Caching

This MVC will act as a pull-through cache of upstream images. Caching was determined to be the first step since it provides a useful foundation that other features (security, rules, etc.) can build upon.

Rails Codebase

The reason we have chosen using the Rails codebase is that it offers a quicker time-to-market for the feature than investigating a different solution immediately, such as Go or other options. The codebase will remain simple enough in the MVC state that we can learn from the feature, get customer feedback, and decide later if a different technology approach is more suitable.

Security

No security scans or support for upstream authenticated repositories is supported yet. It is purely intended to act as an caching proxy server for the defined repositories. These features will arrive in later releases, once we build the functionality that is able to retrieve and cache images locally.

In terms of internal security implementation, we have ensured that the service which is generating requests to upstream servers avoids making requests to the local network. This is similar to all of our other functions which can take a user entered URL, to prevent potential SSRF attacks utilizing the GitLab server. Note that there is a flag to disable these protections, if needed.

Authentication

For the MVC we will not implement authentication, but will also not enable the feature on gitlab.com. Enabling gitlab.com access will come in a follow-up issue.

Group vs. Project vs. Instance Level

The reasons this will be implemented at the group instead of project level are:

  1. We have too much complexity in project itself. If we can unload some features that are necessary only on a group level, we should do it.
  2. Putting the feature into the same UI as a container registry is confusing for a user. You should be able to use a registry proxy even if you don't use a registry itself.
  3. People are likely to create one project per group and re-use it for a whole group as a single proxy endpoint.
  4. In future we introduce dependency proxy for maven and npm. It makes sense to have all dependency proxy features in one place in UI.
  5. Billing is on a group level so it can be used on GitLab.com

Mockup

ee_7934-dependency-proxy-for-container-registries-mvc

Functionality

  • For users that meet the availability standards, navigate to Group home
  • Overview → Dependency Proxy
  • Turn the feature on by toggling the button
  • Click ‘save changes’
  • Copy dependency proxy URL for use from the terminal
  • From the terminal $ docker pull proxy/url/package name version
  • GitLab makes a request to docker hub and will download all blobs that are not stored locally
  • Docker needs:
    • Network connection
    • Will look for latest version
    • Updates to latest version will replace cached version
    • Manifest 1kb text to show dependent blobs isn’t cached
    • Base docker images are often made up of multiple images that will be proxied
  • From the UI you can now see how many blobs are in the proxy and the total size

Instance administrator workflow

There is an admin flag to control whether the proxy feature is available for a given instance. This can be displayed in Admin Area -> CI/CD -> Container Registry for now:

image

The feature is defaulted to on because the endpoint only has access to other public registries and is not a security concern.

For the future, if the project is authenticated, the upstream proxy will also be authenticated to match. So we shouldn't be exposing a new unauthenticated service, for example if all of a given customers projects are private but GitLab is reachable on the internet.

What does success look like, and how can we measure that?

  • We should measure the # of instances with the caching proxy enabled
  • The feature will be utilized by at least X% of people using pipelines
  • We can measure cache hit ratio as a measure of how much this feature is helping
  • Less directly measurable, we expect this feature to drive adoption of Package more generally and even CI/CD for users for whom a proxy was a blocker
  • User engagement in the form of issues/feedback will be a critical success measure of adoption, as with any completely new category

Future Priorities (in priority order)

Links / references

Edited by Tim Rizzi

Linked issues