Registry does not seem to work with a Docker Pass through Proxy and authentication

So, after running into some cross-project headaches with GitLab CI and Registry, I tried to setup the official docker v2 registry in front of the GitLab registry.

From what I read, it should be doable in pass-through-proxy mode. However I've had no luck. I'm not sure if this is an 'issue' and if it is, I'm not sure if it is on GitLab or Registry but thought it would be useful for someone.

I suspect, at the heart of it, there is a authn/authz issue between GitLab Registry and Registry proxy

My setup.

1.) I created a user in my CE omnibus with master permissions on the project in question 2.) I cleaned my local docker credentials and successfully performed 'docker login' with the new user 3.) I successfully pulled an image of the project directly from GitLab

So, at this point I am pretty sure Docker can pull via this new user Next.....

4.) I setup A registry proxy using the user I setup in step 1 (config excerpt below) 5.) log into registry proxy from my local docker client (successful) (using proxy credentials) 6.) pull the same image via registry proxy 7.) Proxy seems to get a JWT Token 8.) Gitlab returns permission denied

Looking at the logs, it looks like I am able to login and get a JWT token. However, I receive a permission denied when fetching the manifest from GitLab.

Short of hacking into the GitLab Auth code, I'm not sure what this is.

This is an onsite install of Omnibus CE

GitLab Logs (of when registry tried to use it)

Started GET "/jwt/auth?scope=repository%3A<group>%2F<project>%3Apull&service=container_registry" for 54.165.165.163 at 2016-06-24 13:20:38 +0000
Processing by JwtController#auth as HTML
Parameters: {"scope"=>"repository:<group>/<project>:pull", "service"=>"container_registry"}
Completed 403 Forbidden in 28ms (Views: 0.2ms | ActiveRecord: 0.7ms)

Registry proxy v2 logs

private-registry_1 | time="2016-06-24T13:28:03.61434415Z" level=debug msg="authorizing request" go.version=go1.6.2 http.request.host=private-mirror.<domain>.com http.request.id=42fdd663-61cd-43ad-90d7-c35bb3fe5f80 http.request.method=GET http.request.remoteaddr=<mirror IP> http.request.uri="/v2/<group>/<project>/manifests/master" http.request.useragent="docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.10.0-327.3.1.el7.x86_64 os/linux arch/amd64" instance.id=3b84c602-24f1-4b61-8e60-7214c4b6848c service=registry vars.name="<group>/<project>" vars.reference=master version=v2.4.1 

private-registry_1 | time="2016-06-24T13:28:03.619664239Z" level=debug msg=GetImageManifest go.version=go1.6.2 http.request.host=private-mirror.<domain>.com http.request.id=42fdd663-61cd-43ad-90d7-c35bb3fe5f80 http.request.method=GET http.request.remoteaddr=<mirror IP> http.request.uri="/v2/<group>/<project>/manifests/master" http.request.useragent="docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.10.0-327.3.1.el7.x86_64 os/linux arch/amd64" instance.id=3b84c602-24f1-4b61-8e60-7214c4b6848c service=registry vars.name="<group>/<project>" vars.reference=master version=v2.4.1 

private-registry_1 | time="2016-06-24T13:28:03.636858307Z" level=info msg="Challenge established with upstream : {https  <nil> registry.<domain>.com /v2/   } map[https://registry.<domain>.com/v2/:[{bearer map[realm:https://gitlab.<domain>.com/jwt/auth service:container_registry]}]]" go.version=go1.6.2 http.request.host=private-mirror.<domain>.com http.request.id=42fdd663-61cd-43ad-90d7-c35bb3fe5f80 http.request.method=GET http.request.remoteaddr=<mirror IP> http.request.uri="/v2/ulq-next/base-api/manifests/master" http.request.useragent="docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.10.0-327.3.1.el7.x86_64 os/linux arch/amd64" instance.id=3b84c602-24f1-4b61-8e60-7214c4b6848c service=registry vars.name="<group>/<project>" vars.reference=master version=v2.4.1 

private-registry_1 | time="2016-06-24T13:28:03.727649698Z" level=debug msg="s3aws.GetContent(\"/docker/registry/v2/repositories/<group>/<project>/_manifests/tags/master/current/link\")" go.version=go1.6.2 http.request.host=private-mirror.<domain>.com http.request.id=42fdd663-61cd-43ad-90d7-c35bb3fe5f80 http.request.method=GET http.request.remoteaddr=<registry IP> http.request.uri="/v2/group/project/manifests/master" http.request.useragent="docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.10.0-327.3.1.el7.x86_64 os/linux arch/amd64" instance.id=3b84c602-24f1-4b61-8e60-7214c4b6848c service=registry trace.duration=42.671889ms trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).GetContent" trace.id=bff24068-3bb1-4c19-abb0-03cc4cf099f5 trace.line=82 vars.name="<group>/<project>" vars.reference=master version=v2.4.1 

private-registry_1 | time="2016-06-24T13:28:03.727852549Z" level=error msg="response completed with error" err.code="manifest unknown" err.detail="unknown tag=master" err.message="manifest unknown" go.version=go1.6.2 http.request.host=private-mirror.<domain>.com http.request.id=42fdd663-61cd-43ad-90d7-c35bb3fe5f80 http.request.method=GET http.request.remoteaddr=<registry ip> http.request.uri="/v2/<group>/<project>/manifests/master" http.request.useragent="docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.10.0-327.3.1.el7.x86_64 os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=115.353074ms http.response.status=404 http.response.written=96 instance.id=3b84c602-24f1-4b61-8e60-7214c4b6848c service=registry vars.name="<group>/<project>" vars.reference=master version=v2.4.1

proxy config segment (Docker Compose)

private-registry:
  image: registry:2
  ports:
    - 127.0.0.1:5001:5000
  environment:
    REGISTRY_LOG_LEVEL: debug
    REGISTRY_PROXY_REMOTEURL: https://registry.<domain.com>.com    #My Gitlab installation
    REGISTRY_PROXY_USERNAME: <gitlab username>
    REGISTRY_PROXY_PASSWORD: <gitlab password>
    REGISTRY_STORAGE: s3
    REGISTRY_STORAGE_S3_REGION: us-east-1
    REGISTRY_STORAGE_S3_BUCKET: <s3 bucket>
    REGISTRY_STORAGE_S3_ROOTDIRECTORY: /registry/mirror/gitlab
    REGISTRY_STORAGE_S3_STORAGECLASS: REDUCED_REDUNDANCY
    REGISTRY_STORAGE_S3_V4AUTH: 'true'
    REGISTRY_STORAGE_S3_ACCESSKEY: <s3 key id>
    REGISTRY_STORAGE_S3_SECRETKEY: <s3 secret>
    REGISTRY_HTTP_SECRET:  <registry http session secret token>
    REGISTRY_STORAGE_CACHE_BLOBDESCRIPTOR: redis
    REGISTRY_REDIS_ADDR: 127.0.0.1:6379
    REGISTRY_REDIS_DB: 1
  volumes:
    - /docker/data:/data
  links:
    - redis:redis
Edited by 🤖 GitLab Bot 🤖