docker push fails - EOF, due to missing registry host configuration
Summary
GitLab Registry subchart (v2.2.12) does not set the registry host in config.yaml
Steps to reproduce
- Helm-based deployment of GitLab v2.2.12
- Set the registry host to be different than that of the unicorn pod (e.g. registry.gitlab.host.com)
-
docker login
to the deployed registry - Attempt to push a container to a valid container registry in GitLab
Configuration used
gitlab:
global:
hosts:
https: true
domain: mygitlab.domain.com
gitlab:
name: app.mygitlab.domain.com
ingress:
enabled: false
# elided for brevity
# ...
registry:
relativeurls: true
log:
level: debug
fields:
service: registry
ingress:
enabled: true
tls:
enabled: false
annotations:
nginx.ingress.kubernetes.io/force-ssl-redirect: false
nginx.ingress.kubernetes.io/enable-cors: true
service:
type: NodePort
storage:
secret: registry-storage
key: config
Current behavior
docker push results in a hang. Docker Daemon logs show
upload failed, retrying: EOF
Expected behavior
docker push to succeed.
Versions
- Chart: 2.2.12
- Platform:
- Cloud: EKS
- Kubernetes:
- Client: 1.14.8
- Server: 1.14.9-eks-c0eccc (v1.14+)
- Helm: (
helm version
)- Client: 2.15.1
- Server: 2.15.1
Relevant logs
docker daemon
2019-12-24T18:22:52Z docker time="2019-12-24T18:22:52.962619447Z" level=debug msg="hostDir: /etc/docker/certs.d/registry.host.com"
2019-12-24T18:22:52Z docker time="2019-12-24T18:22:52.962821176Z" level=debug msg="Trying to push registry.host.com/group/project to https://registry.host.com v2"
2019-12-24T18:22:53Z docker time="2019-12-24T18:22:53.134074334Z" level=debug msg="Pushing repository: registry.host.com/group/project:tank"
2019-12-24T18:23:22Z docker time="2019-12-24T18:23:22.092690728Z" level=debug msg="Pushing layer: sha256:55a513407fe151a93af63e998aca264e30e7d9602fa24bdc1c6abd6956f11f4e"
2019-12-24T18:23:22Z docker time="2019-12-24T18:23:22.273329451Z" level=debug msg="Checking for presence of layer sha256:eac247cb7af5edc34d3620e8bce653d4af7d4e3a0427d487a97530c7fac0b841 (sha256:322973677ef5fb0405c230fed304b2e907f46a1edb921bab269f88547bcf726f) in registry.host.com/group/project"
2019-12-24T18:23:22Z docker time="2019-12-24T18:23:22.713672913Z" level=debug msg="Pushing layer: sha256:eac247cb7af5edc34d3620e8bce653d4af7d4e3a0427d487a97530c7fac0b841"
2019-12-24T18:23:22Z docker time="2019-12-24T18:23:22.857404456Z" level=debug msg="Assembling tar data for b4c6fd9ac2f593d691f900e2ff7b0b5016bff6a73b58646d305c4f06e768cedc"
2019-12-24T18:23:23Z docker time="2019-12-24T18:23:23.050906299Z" level=debug msg="Assembling tar data for fb9f9a525a6bb29c76f89cfff2e73edde01bfc068fbe3bbcd69dc567e8703d67"
2019-12-24T18:33:24Z docker time="2019-12-24T18:33:24.026932821Z" level=error msg="Upload failed, retrying: EOF"
2019-12-24T18:33:24Z docker time="2019-12-24T18:33:24.306256269Z" level=error msg="Upload failed, retrying: EOF"
Registry debug logs
10.100.138.189 - - [24/Dec/2019:18:28:03 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \\(darwin\\))"
time="2019-12-24T18:28:23.567827186Z" level=debug msg="authorizing request" go.version=go1.11.2 http.request.host=registry.gitlab.host.com http.request.id=0563fc67-49a7-4f38-a8e0-ae26b16df54d http.request.method=HEAD http.request.remoteaddr=ZZ.ZZ.ZZ.ZZ http.request.uri="/v2/group/project/blobs/sha256:322973677ef5fb0405c230fed304b2e907f46a1edb921bab269f88547bcf726f" http.request.useragent="docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(darwin\))" vars.digest="sha256:322973677ef5fb0405c230fed304b2e907f46a1edb921bab269f88547bcf726f" vars.name="group/project"
time="2019-12-24T18:28:23.568179331Z" level=info msg="authorized request" go.version=go1.11.2 http.request.host=registry.gitlab.host.com http.request.id=0563fc67-49a7-4f38-a8e0-ae26b16df54d http.request.method=HEAD http.request.remoteaddr=ZZ.ZZ.ZZ.ZZ http.request.uri="/v2/group/project/blobs/sha256:322973677ef5fb0405c230fed304b2e907f46a1edb921bab269f88547bcf726f" http.request.useragent="docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(darwin\))" vars.digest="sha256:322973677ef5fb0405c230fed304b2e907f46a1edb921bab269f88547bcf726f" vars.name="group/project"
time="2019-12-24T18:28:23.568233246Z" level=debug msg=GetBlob auth.user.name=tank go.version=go1.11.2 http.request.host=registry.gitlab.host.com http.request.id=0563fc67-49a7-4f38-a8e0-ae26b16df54d http.request.method=HEAD http.request.remoteaddr=ZZ.ZZ.ZZ.ZZ http.request.uri="/v2/group/project/blobs/sha256:322973677ef5fb0405c230fed304b2e907f46a1edb921bab269f88547bcf726f" http.request.useragent="docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(darwin\))" vars.digest="sha256:322973677ef5fb0405c230fed304b2e907f46a1edb921bab269f88547bcf726f" vars.name="group/project"
time="2019-12-24T18:28:23.626859448Z" level=debug msg="s3aws.GetContent("/docker/registry/v2/repositories/group/project/_layers/sha256/322973677ef5fb0405c230fed304b2e907f46a1edb921bab269f88547bcf726f/link")" auth.user.name=tank go.version=go1.11.2 http.request.host=registry.gitlab.host.com http.request.id=0563fc67-49a7-4f38-a8e0-ae26b16df54d http.request.method=HEAD http.request.remoteaddr=ZZ.ZZ.ZZ.ZZ http.request.uri="/v2/group/project/blobs/sha256:322973677ef5fb0405c230fed304b2e907f46a1edb921bab269f88547bcf726f" http.request.useragent="docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(darwin\))" trace.duration=58.570282ms trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).GetContent" trace.id=6f571cb2-d795-4f6d-8f66-31388666c6bf trace.line=95 vars.digest="sha256:322973677ef5fb0405c230fed304b2e907f46a1edb921bab269f88547bcf726f" vars.name="group/project"
time="2019-12-24T18:28:23.626932243Z" level=error msg="response completed with error" auth.user.name=tank err.code="blob unknown" err.detail=sha256:322973677ef5fb0405c230fed304b2e907f46a1edb921bab269f88547bcf726f err.message="blob unknown to registry" go.version=go1.11.2 http.request.host=registry.gitlab.host.com http.request.id=0563fc67-49a7-4f38-a8e0-ae26b16df54d http.request.method=HEAD http.request.remoteaddr=ZZ.ZZ.ZZ.ZZ http.request.uri="/v2/group/project/blobs/sha256:322973677ef5fb0405c230fed304b2e907f46a1edb921bab269f88547bcf726f" http.request.useragent="docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(darwin\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=59.989956ms http.response.status=404 http.response.written=157 vars.digest="sha256:322973677ef5fb0405c230fed304b2e907f46a1edb921bab269f88547bcf726f" vars.name="group/project"
10.100.138.189 - - [24/Dec/2019:18:28:23 +0000] "HEAD /v2/group/project/blobs/sha256:322973677ef5fb0405c230fed304b2e907f46a1edb921bab269f88547bcf726f HTTP/1.1" 404 157 "" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \\(darwin\\))"
time="2019-12-24T18:28:23.665618691Z" level=debug msg="authorizing request" go.version=go1.11.2 http.request.host=registry.gitlab.host.com http.request.id=f0a248ae-b604-443b-a4ef-a72892f8ec91 http.request.method=POST http.request.remoteaddr=ZZ.ZZ.ZZ.ZZ http.request.uri="/v2/group/project/blobs/uploads/" http.request.useragent="docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(darwin\))" vars.name="group/project"
time="2019-12-24T18:28:23.665939765Z" level=info msg="authorized request" go.version=go1.11.2 http.request.host=registry.gitlab.host.com http.request.id=f0a248ae-b604-443b-a4ef-a72892f8ec91 http.request.method=POST http.request.remoteaddr=ZZ.ZZ.ZZ.ZZ http.request.uri="/v2/group/project/blobs/uploads/" http.request.useragent="docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(darwin\))" vars.name="group/project"
time="2019-12-24T18:28:23.665988394Z" level=debug msg="(*linkedBlobStore).Writer" auth.user.name=tank go.version=go1.11.2 http.request.host=registry.gitlab.host.com http.request.id=f0a248ae-b604-443b-a4ef-a72892f8ec91 http.request.method=POST http.request.remoteaddr=ZZ.ZZ.ZZ.ZZ http.request.uri="/v2/group/project/blobs/uploads/" http.request.useragent="docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(darwin\))" vars.name="group/project"
time="2019-12-24T18:28:23.768025713Z" level=debug msg="s3aws.PutContent("/docker/registry/v2/repositories/group/project/_uploads/3c721ee0-b37b-460f-8c87-c070dc2679a8/startedat")" auth.user.name=tank go.version=go1.11.2 http.request.host=registry.gitlab.host.com http.request.id=f0a248ae-b604-443b-a4ef-a72892f8ec91 http.request.method=POST http.request.remoteaddr=ZZ.ZZ.ZZ.ZZ http.request.uri="/v2/group/project/blobs/uploads/" http.request.useragent="docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(darwin\))" trace.duration=101.982731ms trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).PutContent" trace.id=aff962cd-b97b-4363-a6ff-4a0945fa6efb trace.line=110 vars.name="group/project"
time="2019-12-24T18:28:23.794586721Z" level=debug msg="s3aws.Writer("/docker/registry/v2/repositories/group/project/_uploads/3c721ee0-b37b-460f-8c87-c070dc2679a8/data", false)" auth.user.name=tank go.version=go1.11.2 http.request.host=registry.gitlab.host.com http.request.id=f0a248ae-b604-443b-a4ef-a72892f8ec91 http.request.method=POST http.request.remoteaddr=ZZ.ZZ.ZZ.ZZ http.request.uri="/v2/group/project/blobs/uploads/" http.request.useragent="docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(darwin\))" trace.duration=26.501711ms trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Writer" trace.id=b3a0567e-239e-4ccf-918e-54af045d17d5 trace.line=142 vars.name="group/project"
time="2019-12-24T18:28:23.841034375Z" level=debug msg="s3aws.PutContent("/docker/registry/v2/repositories/group/project/_uploads/3c721ee0-b37b-460f-8c87-c070dc2679a8/hashstates/sha256/0")" auth.user.name=tank go.version=go1.11.2 http.request.host=registry.gitlab.host.com http.request.id=f0a248ae-b604-443b-a4ef-a72892f8ec91 http.request.method=POST http.request.remoteaddr=ZZ.ZZ.ZZ.ZZ http.request.uri="/v2/group/project/blobs/uploads/" http.request.useragent="docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(darwin\))" trace.duration=46.386346ms trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).PutContent" trace.id=22de82b6-79c2-461a-b748-cb6ab5d6cdf0 trace.line=110 vars.name="group/project"
time="2019-12-24T18:28:23.84115734Z" level=info msg="response completed" go.version=go1.11.2 http.request.host=registry.gitlab.host.com http.request.id=f0a248ae-b604-443b-a4ef-a72892f8ec91 http.request.method=POST http.request.remoteaddr=ZZ.ZZ.ZZ.ZZ http.request.uri="/v2/group/project/blobs/uploads/" http.request.useragent="docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(darwin\))" http.response.duration=176.339942ms http.response.status=202 http.response.written=0
10.100.138.189 - - [24/Dec/2019:18:28:23 +0000] "POST /v2/group/project/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \\(darwin\\))"
time="2019-12-24T18:28:23.945071942Z" level=debug msg="authorizing request" go.version=go1.11.2 http.request.host=registry.gitlab.host.com http.request.id=2ebde5ba-663d-4404-9184-6278d311b8b1 http.request.method=POST http.request.remoteaddr=ZZ.ZZ.ZZ.ZZ http.request.uri="/v2/group/project/blobs/uploads/" http.request.useragent="docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(darwin\))" vars.name="group/project"
time="2019-12-24T18:28:23.945411427Z" level=info msg="authorized request" go.version=go1.11.2 http.request.host=registry.gitlab.host.com http.request.id=2ebde5ba-663d-4404-9184-6278d311b8b1 http.request.method=POST http.request.remoteaddr=ZZ.ZZ.ZZ.ZZ http.request.uri="/v2/group/project/blobs/uploads/" http.request.useragent="docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(darwin\))" vars.name="group/project"
time="2019-12-24T18:28:23.945459331Z" level=debug msg="(*linkedBlobStore).Writer" auth.user.name=tank go.version=go1.11.2 http.request.host=registry.gitlab.host.com http.request.id=2ebde5ba-663d-4404-9184-6278d311b8b1 http.request.method=POST http.request.remoteaddr=ZZ.ZZ.ZZ.ZZ http.request.uri="/v2/group/project/blobs/uploads/" http.request.useragent="docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(darwin\))" vars.name="group/project"
time="2019-12-24T18:28:23.9999492Z" level=debug msg="s3aws.PutContent("/docker/registry/v2/repositories/group/project/_uploads/2036195a-0e20-45f3-89c8-52b3e109b501/startedat")" auth.user.name=tank go.version=go1.11.2 http.request.host=registry.gitlab.host.com http.request.id=2ebde5ba-663d-4404-9184-6278d311b8b1 http.request.method=POST http.request.remoteaddr=ZZ.ZZ.ZZ.ZZ http.request.uri="/v2/group/project/blobs/uploads/" http.request.useragent="docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(darwin\))" trace.duration=54.43865ms trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).PutContent" trace.id=c5e1b28b-6c67-4667-8d75-2e89017ccc48 trace.line=110 vars.name="group/project"
time="2019-12-24T18:28:24.05213447Z" level=debug msg="s3aws.Writer("/docker/registry/v2/repositories/group/project/_uploads/2036195a-0e20-45f3-89c8-52b3e109b501/data", false)" auth.user.name=tank go.version=go1.11.2 http.request.host=registry.gitlab.host.com http.request.id=2ebde5ba-663d-4404-9184-6278d311b8b1 http.request.method=POST http.request.remoteaddr=ZZ.ZZ.ZZ.ZZ http.request.uri="/v2/group/project/blobs/uploads/" http.request.useragent="docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(darwin\))" trace.duration=52.137203ms trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Writer" trace.id=cd531802-2902-42ac-8f7c-61411df448ce trace.line=142 vars.name="group/project"
time="2019-12-24T18:28:24.087685642Z" level=debug msg="s3aws.PutContent("/docker/registry/v2/repositories/group/project/_uploads/2036195a-0e20-45f3-89c8-52b3e109b501/hashstates/sha256/0")" auth.user.name=tank go.version=go1.11.2 http.request.host=registry.gitlab.host.com http.request.id=2ebde5ba-663d-4404-9184-6278d311b8b1 http.request.method=POST http.request.remoteaddr=ZZ.ZZ.ZZ.ZZ http.request.uri="/v2/group/project/blobs/uploads/" http.request.useragent="docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(darwin\))" trace.duration=35.485448ms trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).PutContent" trace.id=a9ed7e5c-c150-48f3-a357-e9615b3608ec trace.line=110 vars.name="group/project"
time="2019-12-24T18:28:24.087813825Z" level=info msg="response completed" go.version=go1.11.2 http.request.host=registry.gitlab.host.com http.request.id=2ebde5ba-663d-4404-9184-6278d311b8b1 http.request.method=POST http.request.remoteaddr=ZZ.ZZ.ZZ.ZZ http.request.uri="/v2/group/project/blobs/uploads/" http.request.useragent="docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(darwin\))" http.response.duration=144.39219ms http.response.status=202 http.response.written=0
10.100.138.189 - - [24/Dec/2019:18:28:23 +0000] "POST /v2/group/project/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \\(darwin\\))"
After a lengthy exchange and much digging, this behaviour was addressd by configuring the hostname on the registry. The existing chart v2.2.12 does not have an entry in the registry configmap for host. When I added an entry for host under the second http block after helm template
, (why are there two here?), the docker push
succeeded.
The host could be read out of the globals, similar to how it can be set for gitlab
, or it could be constructed using registry.$DOMAIN
, or something like that.
For reference, the GitLab Support ticket number is 141800