Skip to content

Add project and root IDs to registry JWT tokens metadata

João Pereira requested to merge registry-jwt-ids into master

What does this MR do and why?

Related to https://gitlab.com/gitlab-org/container-registry/-/issues/1214. Note that the linked issue is confidential only because the parent epic needs to be, but there is no confidentiality around this specific change.

In order to provide additional insight into the container registry data transfers, we need to collect the GitLab project ID and the GitLab top-level group/namespace ID that a given registry request relates to.

This MR updates the Container Registry JWT tokens emitted by Rails so that they include new project_id and root_namespace_id metadata attributes, expanding the list of existing metadata attributes (project_path) already included there.

This metadata will then be parsed by the registry on a future iteration and used for metrics purposes.

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

How to set up and validate locally

We'll assume the existence of the gitlab-org/gitlab-test project in your GDK instance and use it for these tests. You can pick any other project, as long as the Container Registry feature is enabled for it.

We'll also assume that your GDK Rails instance is listening at gdk.test:3000 and you have a user root with a PAT foobar, with read permissions on the container registry. Your GDK registry setup must have auth_enabled: true (docs).

We need to query the Rails API and decode a few JWT tokens during these tests. To do so you can either decode it with a JWT debugger on the web (e.g. https://jwt.io/) or use a CLI tool such as jwt-cli. We'll use the latter. We'll also use httpie to make the requests and jq to parse the token from the response, but you can do that with any other tools or manually.

Current behavior

  1. Checkout the master branch of this repository.

  2. Open up a Rails console and a terminal window/pane.

  3. On your terminal, obtain a JWT token with e.g. pull permissions for repository gitlab-org/gitlab-test/foo. This will test the flow for external authentication requests:

    http -a root:foobar http://gdk.test:3000/jwt/auth \
    client_id=="docker" \
    service=="container_registry" \
    scope=="repository:gitlab-org/gitlab-test/foo:pull" | jq -r '.token' | jwt decode -

    Output (timestamp fields will vary, the important part is access):

    Token header
------------
...

Token claims
------------
 {
   "access": [
     {
       "actions": [
         "pull"
       ],
       "meta": {
         "project_path": "gitlab-org/gitlab-test"
       },
       "name": "gitlab-org/gitlab-test/foo",
       "type": "repository"
     }
   ],
   "aud": "container_registry",
   "auth_type": "personal_access_token",
   "exp": 1710868064,
   "iat": 1710867764,
   "iss": "gitlab-issuer",
   "jti": "d3fbb723-6207-45e2-9eb6-40ce8c1f3f7b",
   "nbf": 1710867759,
   "sub": "root",
   "user": "eyJraWQiOiI0NUVXOjQ2Uzc6WlZBRzpKTFpBOkNUTFY6V1Q0NzpZQkJLOkZZVEk6VURRTzpXVkFCOktRVlQ6UlhOQiIsInR5cCI6IkpXVCIsImFsZyI6IlJTMjU2In0.eyJ1c2VyX2luZm8iOnsidG9rZW5fdHlwZSI6InBlcnNvbmFsX2FjY2Vzc190b2tlbiIsInVzZXJuYW1lIjoicm9vdCIsInVzZXJfaWQiOjF9LCJqdGkiOiI5YmY2MDg0Ny0wMzIwLTRiYjYtYWI4Ni1lMzMxZDRjYWI0ZTciLCJpYXQiOjE3MTA4Njc3NjQsIm5iZiI6MTcxMDg2Nzc1OSwiZXhwIjoxNzEwODY3ODI0fQ.xq_6sBGNcn7JNjVzEu5mu1PrEcQxn-wzFumxWrU26BIWbSMIfcBLdjyB45eE25II0BUpRFe19PcdDo93_TnRf86OzNZiFmfuyFdhORS9lLRaTbJSX35QTr9ztkd7iA84xaKzZbTumV29VrZCqs_JumqfEFxXlRYQxvPn_-ukSZnZYqB1vCDQnEYL77C5TJ8lYeX-ipxTHTTEn-2YNCvtNzpnddGWrezkQe29A7yxgNrsYWCZ43RyIx2NVrm7kWxQMKCCgLXcTEMG3chdMgEH9LEBaS9eP6tzUNRebnBf_NBLMJN-W-Roci7GdWcDbO803Oo09ibDD4HrPbL8Mkc1EA"
 }

  4. On the Rails console, go ahead and request an e.g. full_access_token for the same repository. This will simulate the emission of a token for internal (i.e. Rails backend) use:

    [1] pry(main)> JWT.decode(Auth::ContainerRegistryAuthenticationService.full_access_token('gitlab-org/gitlab-test/foo'), nil, false)
=> [{"access"=>[{"type"=>"repository", "name"=>"gitlab-org/gitlab-test/foo", "actions"=>["*"], "meta"=>{"project_path"=>"gitlab-org/gitlab-test"}}], "jti"=>"342038b6-a70f-4fdb-a338-a92e03a1d057", "aud"=>"container_registry", "iss"=>"gitlab-issuer", "iat"=>1710867791, "nbf"=>1710867786, "exp"=>1710868091}, {"kid"=>"45EW:46S7:ZVAG:JLZA:CTLV:WT47:YBBK:FYTI:UDQO:WVAB:KQVT:RXNB", "typ"=>"JWT", "alg"=>"RS256"}]

New behavior

  1. Check out this MR's branch.

  2. Repeat the /jwt/auth request. This time we see the new meta object and its project_id and root_namespace_id attributes being filled:

    Token header
------------
...

Token claims
------------
{
  "access": [
    {
      "actions": [
        "pull"
      ],
      "meta": {
        "project_id": 2,
        "project_path": "gitlab-org/gitlab-test",
        "root_namespace_id": 24
      },
      "name": "gitlab-org/gitlab-test/foo",
      "type": "repository"
    }
  ],
  "aud": "container_registry",
  "auth_type": "personal_access_token",
  "exp": 1710867763,
  "iat": 1710867463,
  "iss": "gitlab-issuer",
  "jti": "22879db5-2727-4c06-b7fe-5bff3a1f4b1c",
  "nbf": 1710867458,
  "sub": "root",
  "user": "eyJraWQiOiI0NUVXOjQ2Uzc6WlZBRzpKTFpBOkNUTFY6V1Q0NzpZQkJLOkZZVEk6VURRTzpXVkFCOktRVlQ6UlhOQiIsInR5cCI6IkpXVCIsImFsZyI6IlJTMjU2In0.eyJ1c2VyX2luZm8iOnsidG9rZW5fdHlwZSI6InBlcnNvbmFsX2FjY2Vzc190b2tlbiIsInVzZXJuYW1lIjoicm9vdCIsInVzZXJfaWQiOjF9LCJqdGkiOiIxMDQ0ZjI0Ni1kMDVlLTRhNDktOWFhMC1hM2Q1Yjg3YzJlOTMiLCJpYXQiOjE3MTA4Njc0NjMsIm5iZiI6MTcxMDg2NzQ1OCwiZXhwIjoxNzEwODY3NTIzfQ.AoqyLdcCNrp4VyAcEwuEik19uielaRC_Zy2mHFJv5yToOwGsp8Fa7cgF2HeLZmUUM7bjv1S0OFTkadkO6Jljmodl2W6KPTDErTpq1ocvVvJus5M9SKdO3mymKEj9q390tDAeVstxHtESTZzkaIoVt_B45Y0wDuEMA26yqmMa2BYBRyxzgaJMeE1zuz09sXPgVJ5Ho-jJg4LY0gK7wA7QRY73-dhKd7tPff_tXfw0Plq5lIq5V6bkY8I7-3Hv9N8YSpMEnXU5rlM2EJj9OKfJKAysy3yACIbOXHTn9aoBIW18uG54BiO34Yb0wKQtPxcocyJa5MB3Ho94imJguJbhJg"
}

  3. Do the same on the Rails console:

    [1] pry(main)> JWT.decode(Auth::ContainerRegistryAuthenticationService.full_access_token('gitlab-org/gitlab-test/foo'), nil, false)
=> [{"access"=>[{"type"=>"repository", "name"=>"gitlab-org/gitlab-test/foo", "actions"=>["*"], "meta"=>{"project_path"=>"gitlab-org/gitlab-test", "project_id"=>2, "root_namespace_id"=>24}}], "jti"=>"2381a571-3d44-4873-b8de-6f756c278f1a", "aud"=>"container_registry", "iss"=>"gitlab-issuer", "iat"=>1710867690, "nbf"=>1710867685, "exp"=>1710867990, {"kid"=>"45EW:46S7:ZVAG:JLZA:CTLV:WT47:YBBK:FYTI:UDQO:WVAB:KQVT:RXNB", "typ"=>"JWT", "alg"=>"RS256"}]

  4. You can also check that the registry is OK with receiving the modifying tokens (unknown attributes are ignored) by e.g. listing the tags of a given container repository, which internally will trigger the generation of a full_access_token and send a request to the registry with it:

    [7] pry(main)> repo = ContainerRepository.find_by_path(ContainerRegistry::Path.new('gitlab-org/gitlab-test/foo'))
=> #<ContainerRepository:0x000000014354b9f8
  id: 5,
  project_id: 2,
  name: "foo",
  created_at: Wed, 15 Mar 2023 13:00:17.206705000 UTC +00:00,
  updated_at: Wed, 15 Mar 2023 13:00:17.206705000 UTC +00:00,
  status: nil,
  expiration_policy_started_at: Mon, 13 Mar 2023 15:11:37.871130000 UTC +00:00,
  expiration_policy_cleanup_status: "cleanup_unfinished",
  expiration_policy_completed_at: nil,
  migration_pre_import_started_at: nil,
  migration_pre_import_done_at: nil,
  migration_import_started_at: nil,
  migration_import_done_at: nil,
  migration_aborted_at: nil,
  migration_skipped_at: nil,
[8] pry(main)> repo.tags.count
=> 24
Edited by João Pereira

Merge request reports