Skip to content

Allow gradual rollout for the container registry Google CDN feature

João Pereira requested to merge container_registry_cdn_redirect into master

What does this MR do and why?

As part of https://gitlab.com/groups/gitlab-org/-/epics/7212, we added support for Google Cloud CDN to the Container Registry (https://gitlab.com/gitlab-org/container-registry/-/issues/535).

This MR introduces a new feature flag (container_registry_cdn_redirect) and modifies the container registry auth service to allow for a gradual percentage-based rollout of the new Google Cloud CDN feature. Please see the description of https://gitlab.com/gitlab-org/gitlab/-/issues/349417 for context, technical background and the rationale behind this change.

Notes for Security Review

This MR does not make any changes related to the evaluation of credentials and/or permissions. It simply uses the JWT tokens as a medium to communicate a feature flag value to the Container Registry. This is the second time we do something like this, the first one was in !63907 (merged).

Also, this MR is only for the feature flag. The readiness review (including the security review) of the actual CDN feature will be conducted as part of https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/14903.

How to set up and validate locally

We'll assume the existence of the gitlab-org/gitlab-test project in your GDK instance and use it for these tests. You can pick any other project, as long as the Container Registry feature is enabled for it.

We'll also assume that your GDK Rails instance is listening at gdk.test:3000 and you have a user root with a PAT foobar, with reads permissions to the container registry.

We need to query the Rails API and decode a few JWT tokens during these tests. To do so you can either decode it with a JWT debugger on the web (e.g. https://jwt.io/) or use a CLI tool such as jwt-cli. We'll use the latter. We'll also use httpie to make the requests and jq to parse the token from the response, but you can do that with any other tools or manually.

Current behavior

  1. Checkout the master branch of this repository.

  2. Open up a Rails console and a terminal.

  3. On your terminal, obtain a JWT token for gitlab-org/gitlab-test:

    http -a 'root:foobar' http://gdk.test:3000/jwt/auth \
    client_id=="docker" \
    service=="container_registry" \
    scope=="repository:gitlab-org/gitlab-test:pull" | jq -r '.token' | jwt decode -

    Output (timestamp fields will vary, the important part is access):

    Token header
------------
...

Token claims
------------
{
  "access": [
    {
      "actions": [
        "pull"
      ],
      "name": "gitlab-org/gitlab-test",
      "type": "repository"
    }
  ],
  "aud": "container_registry",
  "exp": 1641558105,
  "iat": 1641557805,
  "iss": "gitlab-issuer",
  "jti": "51d7aa5d-5077-4d27-94fc-a250f4e7501e",
  "nbf": 1641557800,
  "sub": "root"
}

New behavior

  1. Checkout this MR's branch.

  2. Repeat the /jwt/auth request. The output should be identical to the above (except for the timestamp fields) because the feature flag is disabled by default.

  3. Enable container_registry_cdn_redirect for 50% of the time:

    Feature.enable_percentage_of_time(:container_registry_cdn_redirect, 50)

  4. Repeat the /jwt/auth request multiple times. After a couple of retries you should see a modified token with the new cdn_redirect field:

    Token header
------------
...

Token claims
------------
{
  "access": [
    {
      "actions": [
        "pull"
      ],
      "cdn_redirect": true,
      "name": "gitlab-org/gitlab-test",
      "type": "repository"
    }
  ],
  "aud": "container_registry",
  "exp": 1641558504,
  "iat": 1641558204,
  "iss": "gitlab-issuer",
  "jti": "c750d758-8944-4a86-ab34-bca6082ce2ee",
  "nbf": 1641558199,
  "sub": "root"
}

  5. Enable container_registry_cdn_redirect globally:

    Feature.enable(:container_registry_cdn_redirect)

  6. Repeat the /jwt/auth request. After a while (takes a bit of time to propagate), all emitted tokens should now have the cdn_redirect flag.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by João Pereira

Merge request reports