Skip to content

docker-registry not usable with IRSA (or it's not obvious how to do this)

Summary

I've deployed the latest version of gitlab, including the docker registry, on AWS using IAM Roles for Service Accounts (IRSA) as dealt with in gitlab-org/charts/gitlab#2683 (closed).

The pod basically can access the bucket using the go SDK without any special configuration - I verified this by executing a simple go application and listing the bucket contents.

Current versions of the docker registry seem to handle this automatically (see https://github.com/distribution/distribution/blob/main/registry/storage/driver/s3-aws/s3.go#L398), but the version used by gitlab is considerably older (compare the function) and appears not to support IRSA, or at least not without some additional configuration.

Steps to reproduce

Deploy the gitlab docker registry on EKS with a corresponding IAM role configured, try to access it using curl -X GET --header "Authorization: Bearer $TOKEN" https://registry.xxxxxxxxxx.net/v2/_catalog.

Configuration used

Helm chart:

      - global:
          serviceAccount:  
            enabled: true
            create: true  
            annotations:
              eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxxxx:role/gitlab-role
        registry:
          image:
            tag: 'v3.4.1-gitlab'      
          enabled: true
          storage:
            secret: gitlab-registry-s3-config
          log:
            level: debug

And a corresponding secret:

---
apiVersion: v1
kind: Secret
metadata:
  name: gitlab-registry-s3-config
  namespace: gitlab
type: Opaque
data:
  config: |
    s3:
      bucket: xxxxxxxxxxx-gitlab-docker-registry
      region: eu-central-1

Current behavior

The docker registry cannot access the bucket.

Expected behavior

The docker registry should be able to access the bucket.

Versions

  • Platform:
    • Cloud: EKS

Relevant logs

gitlab-registry-6678dccc79-q2skr:registry {"correlation_id":"01F5ZJTPWDWAM9VC7CC5THPTG3","go_version":"go1.16.3","level":"debug","msg":"authorizing request","root_repo":"","time":"2021-05-18T11:12:29Z"}
gitlab-registry-6678dccc79-q2skr:registry {"auth_user_name":"<username>","correlation_id":"01F5ZJTPWDWAM9VC7CC5THPTG3","go_version":"go1.16.3","level":"info","msg":"authorized request","root_repo":"","time":"2021-05-18T11:12:29Z"}                                                                                                                                                                                              
gitlab-registry-6678dccc79-q2skr:registry {"auth_user_name":"<username>","correlation_id":"01F5ZJTPWDWAM9VC7CC5THPTG3","go_version":"go1.16.3","level":"debug","msg":"s3aws.ListObjectsV2Pages(docker/registry/v2/repositories/)","root_repo":"","time":"2021-05-18T11:12:29Z","trace_duration":34584157,"trace_file":"/go/src/github.com/docker/distribution/registry/storage/driver/s3-aws/s3.go","trace_func":"github.com/docker/distribution/registry/storage/driver/s3-aws.(*driver).doWalk","trace_id":"ceec56fe-e153-4047-aba4-2be6fb086722","trace_line":1266,"trace_parent_id":"77402e09-a7ca-4703-9164-9bb8da3f82b6"}                 
gitlab-registry-6678dccc79-q2skr:registry {"auth_user_name":"<username>","correlation_id":"01F5ZJTPWDWAM9VC7CC5THPTG3","go_version":"go1.16.3","level":"debug","msg":"s3aws.Walk(\"/docker/registry/v2/repositories\")","root_repo":"","time":"2021-05-18T11:12:29Z","trace_duration":34654572,"trace_file":"/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go","trace_func":"github.com/docker/distribution/registry/storage/driver/base.(*Base).Walk","trace_id":"77402e09-a7ca-4703-9164-9bb8da3f82b6","trace_line":232}                                                                                           
gitlab-registry-6678dccc79-q2skr:registry {"auth_user_name":"<username>","code":"UNKNOWN","correlation_id":"01F5ZJTPWDWAM9VC7CC5THPTG3","detail":"s3aws: AccessDenied: Access Denied\n\tstatus code: 403, request id: RQ2FP8DZ59BWW5R3, host id: pAtJ1mVmHLU3NqhJ/l89EiDy/4SOz8S21alubwtaDLictJnfXPLqh4rVLnzCLW3fGL8tQ5lgo2k=","error":"unknown: unknown error","go_version":"go1.16.3","level":"error","msg":"unknown error","root_repo":"","time":"2021-05-18T11:12:29Z"}
Edited by Benjamin Goose