Skip to content

Update request auhenticator for the dependency proxy routes

David Fernandez requested to merge 338925-update-rack-attack-for-the-dependency-proxy into master

🚏 Context

The dependency proxy is a way for users to use GitLab as a pull through cache for DockerHub images. In other words, users can pull DockerHub container images through GitLab.

Here is a simplified schema for the interactions

sequenceDiagram
    autonumber
    $ docker->>GitLab JWT service: Hello, here are my credentials, give me a JWT token
    GitLab JWT service -->> $ docker: Certainly. Here is your JWT token
    $ docker->>GitLab dependency proxy: Here is my JWT token, give the manifest file for `alpine:latest`
    GitLab dependency proxy -->> $ docker: Certainly. Here is your manifest
    $ docker->>GitLab dependency proxy: Here is my JWT token, give the blob content for sha X
    GitLab dependency proxy -->> $ docker: Certainly. Here is your blob content

A few notes:

  • A container image can contain more than one layer. As such, interactions (5.) and (6.) can be repeated by the $ docker client.
  • The important thing here is that requests (3.) and (5.) are authenticated by a JWT token.

On the other hand, we have rack-attack enabled on GitLab. We have a few throttles defined.

In #338925 (closed), it has been noted that requests (3.) and (5.) are matched by the unauthenticated web throttle which has a quite low rate limit. The problem is that, as noted, $ docker could fire many requests (5.) in a very short period of time. This can lead rack-attack to reject requests (5.) with 429 Too Many Requests.

This MR aims to update the request authenticator used by rack-attack to properly read the jwt token on requests (3.) and (5.) in order to extract the user and so, get those requests identified by the authenticated web throttle (which usually has a higher rate limit than unauthenticated web)

🔬 What does this MR do and why?

  • Update the request authenticator so that dependency proxy request are properly identified and a user is extracted from requests to manifests/blobs.
  • Refactor DependencyProxy::AuthTokenService so that it's easily re-usable.
    • Basically let it return the user.
  • Update related specs

📸 Screenshots or screen recordings

Setup

To check which throttles are evaluated by Rack attack, I simply added some logger #debug statements:

$ g diff
diff --git a/lib/gitlab/rack_attack.rb b/lib/gitlab/rack_attack.rb
index 3f4c0fa45aa..18deb4fef85 100644
--- a/lib/gitlab/rack_attack.rb
+++ b/lib/gitlab/rack_attack.rb
@@ -102,6 +102,7 @@ def self.configure_throttles(rack_attack)
 
       throttle_or_track(rack_attack, 'throttle_unauthenticated_web', Gitlab::Throttle.unauthenticated_web_options) do |req|
         if req.throttle_unauthenticated_web?
+          Rails.logger.debug('🕵️' * 90)
           req.ip
         end
       end
@@ -117,6 +118,7 @@ def self.configure_throttles(rack_attack)
 
       throttle_or_track(rack_attack, 'throttle_authenticated_web', Gitlab::Throttle.authenticated_web_options) do |req|
         if req.throttle_authenticated_web?
+          Rails.logger.debug('🦊' * 90)
           req.throttled_user_id([:api, :rss, :ics])
         end
       end

(who said I can't use emojis in debugging logs)

From there, I just tail the development.log file:

  • If a 🕵️ appears, that means that throttle unauthenticated_web has been evaluated.
  • If a 🦊 appears, that means that throttle authenticated_web has been evaluated.

For the container image to be pulled, I will use alpine:latest. This image is quite small and at the time of this writing, pulling this image will require:

  • 1 GET manifest request
  • 2 GET blob requests

Personal Access Token

  • Docker logout from the gitlab url 
    $ docker logout gdk.test:8000
  • Docker login with the username and personal access token 
    $ docker login gdk.test:8000
  • Destroy the alpine image for the given group locally (using the Docker dashboard for example)
  • Pull the alpine through the dependency proxy 
    $ docker pull gdk.test:8000/<group_path>/dependency_proxy/containers/alpine:latest

Rack attack throttles evaluated:

action master this MR
GET manifest unauthenticated_web + authenticated_web authenticated_web
GET blob unauthenticated_web + authenticated_web authenticated_web

Deploy Token

  • Docker logout from the gitlab url 
    $ docker logout gdk.test:8000
  • Docker login with the username and personal access token 
    $ docker login gdk.test:8000
  • Destroy the alpine image for the given group locally (using the Docker dashboard for example)
  • Pull the alpine through the dependency proxy 
    $ docker pull gdk.test:8000/<group_path>/dependency_proxy/containers/alpine:latest

Rack attack throttles evaluated:

action master this MR
GET manifest unauthenticated_web + authenticated_web authenticated_web
GET blob unauthenticated_web + authenticated_web authenticated_web

⚠ Regarding deploy tokens, please have a look at #342481 (closed).

CI job Token

  • Destroy the alpine image for the given group locally (using the Docker dashboard for example)
  • Start a new pipeline from the project (see below to setup such project)

Rack attack throttles evaluated:

action master this MR
GET manifest unauthenticated_web + authenticated_web authenticated_web
GET blob unauthenticated_web + authenticated_web authenticated_web

Conclusions

On master, both throttles unauthenticated_web and authenticated_web are evaluated. Because unauthenticated_web is evaluated, the dependency proxy interactions can trigger 429 Too Many Requests responses.

With this MR, the only throttle evaluated for manifests/blobs pulls is authenticated_web. Because that throttle has (usually) a higher rate than unauthenticated_web, it's less likely to see a 429 Too Many Requests

🚠 How to set up and validate locally

Requirements:

  • A group (whatever visibility)
  • A project in that group
    • Create a .gitlab-ci.yml with this content: 
      image: ${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/alpine:latest

stages:
  - deploy

deploy:
  stage: deploy
  script:
    - echo "bananas!"
  • runner properly setup
  • Enable the registry

Steps

  1. Setup throttles in a rails console (⚠ the unauthenticated throttle is a super hard one. Don't forget to either disable that throttle or put back the old values after testing this) 
       Gitlab::CurrentSettings.current_application_settings.update(throttle_unauthenticated_enabled: true, throttle_unauthenticated_period_in_seconds: 10, throttle_unauthenticated_requests_per_period: 2, throttle_authenticated_web_enabled:true, throttle_authenticated_web_period_in_seconds: 10, throttle_authenticated_web_requests_per_period: 100)
  2. Docker login with the credentials of your choice (pat or deploy token) 
    $ docker login gdk.test:8000
  3. Pull an image through the dependency proxy: 
    $ docker pull gdk.test:8000/<group_path>/dependency_proxy/containers/alpine:latest

This last step

  • On master, the pull will hit 429 Too Many Requests 💥 
    $ docker pull gdk.test:8000/bananas1/dependency_proxy/containers/alpine:latest
Error response from daemon: error parsing HTTP 429 response body: invalid character 'R' looking for beginning of value: "Retry later\n"
  • With this MR, the pull will get through ✅ 
    $ docker pull gdk.test:8000/bananas1/dependency_proxy/containers/alpine:latest
latest: Pulling from bananas1/dependency_proxy/containers/alpine
a0d0a0d46f8b: Pull complete 
Digest: sha256:69704ef328d05a9f806b6b8502915e6a0a4faa4d72018dc42343f511490daf8a
Status: Downloaded newer image for gdk.test:8000/bananas1/dependency_proxy/containers/alpine:latest
gdk.test:8000/bananas1/dependency_proxy/containers/alpine:latest

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by David Fernandez

Merge request reports