Can not push docker image to gitlab-registry

Summary

After building the gitlab with the official cloud helm chart.

Trying to push image with docker push, but get "error parsing HTTP 403 response body: unexpected end of JSON input: "

Steps to reproduce

docker push registry.XXXXXX.com/jz21/libreoffice/online
The push refers to repository [registry.XXXXXX.com/jz21/libreoffice/online]
24ba5d55b354: Pushing   2.56kB
b4ab9bd0196c: Pushing   2.56kB
5cddb9c7b614: Pushing [=>                                                 ]  4.974MB/157.1MB
54a1ab664caf: Pushing   2.56kB
6eab474e868e: Pushing  2.048kB
291cccba4a66: Waiting
d6b0e2db295d: Waiting
7d753ddcb211: Waiting
45a0b9c64e4b: Waiting
4bb742556520: Waiting
eba98f08e19d: Waiting
35ff711430e9: Waiting
7bd3fc0b67a7: Waiting
e56e75e05c6e: Waiting
fc3dbf48eb87: Waiting
8d3c56d5f03a: Waiting
93bc1bacb38c: Waiting
afa0ed7bc51b: Waiting
740b0228ee5c: Waiting
bf3d982208f5: Waiting
cd7b4cc1c2dd: Waiting
3a0404adc8bd: Waiting
82718dbf791d: Waiting
c8aa3ff3c3d3: Waiting
error parsing HTTP 403 response body: unexpected end of JSON input: ""

Configuration used

using ceph as S3 storage

here is the ceph storage

s3:
  accesskey: XXXXXXXXXXXXXXXXXXXXX
  secretkey: XXXXXXXXXXXXXXXXXXXXXXXXXXXX
  bucket: Gitlab-registry
  region: optional
  regionendpoint: http://ceph.XXX.org:7480
  secure: false
  v4auth: true
  rootdirectory: /
cache:
  blobdescriptor: 'inmemory'
delete:
  enabled: true

Using my own nginx-ingress controller, already set http2 to off

  config: 
    ssl-ciphers: "ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA"
    ssl-protocols: "TLSv1 TLSv1.1 TLSv1.2"
    hsts-include-subdomains: "false"
    server-name-hash-bucket-size: "256"
    enable-vts-status: "true"
    use-http2: "false"
    server-tokens: "false"

What I found

https://gitlab.com/gitlab-org/gitlab-ce/issues/19356#note_12801594

Here it said need to set

registry_nginx['custom_gitlab_server_config'] = 'proxy_cache_convert_head off;'

But I dont know where to set this config, also tried to check with the ingress annotation, but cannot find related setting also

Also found issue #707 (closed), said to turn off http2, tried but still the same.

Versions

  • Chart: v1.02 app: 11.2.3
  • Platform:
    • Self-hosted: Kubernetes on CoreOS
  • Kubernetes: (kubectl version)
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.7", GitCommit:"dd5e1a2978fd0b97d9b78e1564398aeea7e7fe92", GitTreeState:"clean", BuildDate:"2018-04-19T00:05:56Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.7+coreos.0", GitCommit:"b53ce5708912bcc0a75e8da77e3124e3c8691229", GitTreeState:"clean", BuildDate:"2018-04-23T16:56:24Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
  • Helm: (helm version)
Client: &version.Version{SemVer:"v2.10.0", GitCommit:"9ad53aac42165a5fadc6c87be0dea6b115f93090", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.10.0", GitCommit:"9ad53aac42165a5fadc6c87be0dea6b115f93090", GitTreeState:"clean"}

Relevant logs

10.2.42.0 - - [22/Sep/2018:08:11:42 +0000] "HEAD /v2/jz21/libreoffice/online/blobs/sha256:83eef249e72bc52f6a8d0e45713cbbacd8313c4341b2ed70ea98f05cd3e0c6c7 HTTP/1.1" 404 157 "" "docker/18.06.1-ce go/go1.10.3 git-commit/e68fc7a kernel/4.9.93-linuxkit-aufs os/linux arch/amd64 UpstreamClient(Docker-Client/18.06.1-ce \\(darwin\\))"
time="2018-09-22T08:11:42.809277321Z" level=error msg="response completed with error" auth.user.name=jz21 err.code="blob unknown" err.detail=sha256:ada31e9d79183fca354f5b9df0ce8e6c9411c9ab80e8d9941353b49dfeaec358 err.message="blob unknown to registry" go.version=go1.7.6 http.request.host=registry.XXXXXX.com http.request.id=f31e58d1-d01f-45cf-b092-0bc3451d4c5d http.request.method=HEAD http.request.remoteaddr=172.16.164.118 http.request.uri="/v2/jz21/libreoffice/online/blobs/sha256:ada31e9d79183fca354f5b9df0ce8e6c9411c9ab80e8d9941353b49dfeaec358" http.request.useragent="docker/18.06.1-ce go/go1.10.3 git-commit/e68fc7a kernel/4.9.93-linuxkit-aufs os/linux arch/amd64 UpstreamClient(Docker-Client/18.06.1-ce \\(darwin\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=5.699436ms http.response.status=404 http.response.written=157 instance.id=ddd2d3be-a3a2-4775-b468-131345a1e85d service=registry vars.digest="sha256:ada31e9d79183fca354f5b9df0ce8e6c9411c9ab80e8d9941353b49dfeaec358" vars.name="jz21/libreoffice/online" version=v2.6.2
10.2.6.0 - - [22/Sep/2018:08:11:42 +0000] "HEAD /v2/jz21/libreoffice/online/blobs/sha256:ada31e9d79183fca354f5b9df0ce8e6c9411c9ab80e8d9941353b49dfeaec358 HTTP/1.1" 404 157 "" "docker/18.06.1-ce go/go1.10.3 git-commit/e68fc7a kernel/4.9.93-linuxkit-aufs os/linux arch/amd64 UpstreamClient(Docker-Client/18.06.1-ce \\(darwin\\))"
time="2018-09-22T08:11:42.811578164Z" level=error msg="response completed with error" auth.user.name=jz21 err.code="blob unknown" err.detail=sha256:1b8f2244dc1b5681b3e494c091a8ca61b2c28f6bc8dc37ee3aa38cd5070bf234 err.message="blob unknown to registry" go.version=go1.7.6 http.request.host=registry.XXXXXX.com http.request.id=1b59e576-b65c-4ec0-8aae-6dcbbb2ac600 http.request.method=HEAD http.request.remoteaddr=172.16.164.118 http.request.uri="/v2/jz21/libreoffice/online/blobs/sha256:1b8f2244dc1b5681b3e494c091a8ca61b2c28f6bc8dc37ee3aa38cd5070bf234" http.request.useragent="docker/18.06.1-ce go/go1.10.3 git-commit/e68fc7a kernel/4.9.93-linuxkit-aufs os/linux arch/amd64 UpstreamClient(Docker-Client/18.06.1-ce \\(darwin\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=7.644127ms http.response.status=404 http.response.written=157 instance.id=ddd2d3be-a3a2-4775-b468-131345a1e85d service=registry vars.digest="sha256:1b8f2244dc1b5681b3e494c091a8ca61b2c28f6bc8dc37ee3aa38cd5070bf234" vars.name="jz21/libreoffice/online" version=v2.6.2
10.2.18.0 - - [22/Sep/2018:08:11:42 +0000] "HEAD /v2/jz21/libreoffice/online/blobs/sha256:1b8f2244dc1b5681b3e494c091a8ca61b2c28f6bc8dc37ee3aa38cd5070bf234 HTTP/1.1" 404 157 "" "docker/18.06.1-ce go/go1.10.3 git-commit/e68fc7a kernel/4.9.93-linuxkit-aufs os/linux arch/amd64 UpstreamClient(Docker-Client/18.06.1-ce \\(darwin\\))"
10.2.42.0 - - [22/Sep/2018:08:11:43 +0000] "POST /v2/jz21/libreoffice/online/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/18.06.1-ce go/go1.10.3 git-commit/e68fc7a kernel/4.9.93-linuxkit-aufs os/linux arch/amd64 UpstreamClient(Docker-Client/18.06.1-ce \\(darwin\\))"
10.2.6.0 - - [22/Sep/2018:08:11:43 +0000] "POST /v2/jz21/libreoffice/online/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/18.06.1-ce go/go1.10.3 git-commit/e68fc7a kernel/4.9.93-linuxkit-aufs os/linux arch/amd64 UpstreamClient(Docker-Client/18.06.1-ce \\(darwin\\))"
time="2018-09-22T08:11:44.960280372Z" level=error msg="response completed with error" auth.user.name=jz21 err.code="blob unknown" err.detail=sha256:5aff3f5eea5bc757acfbd17cf27c4dd136890d4f82ef3541db410b222cefe8ec err.message="blob unknown to registry" go.version=go1.7.6 http.request.host=registry.XXXXXX.com http.request.id=899aa730-6c11-443b-bafc-414b335afb58 http.request.method=HEAD http.request.remoteaddr=172.16.164.118 http.request.uri="/v2/jz21/libreoffice/online/blobs/sha256:5aff3f5eea5bc757acfbd17cf27c4dd136890d4f82ef3541db410b222cefe8ec" http.request.useragent="docker/18.06.1-ce go/go1.10.3 git-commit/e68fc7a kernel/4.9.93-linuxkit-aufs os/linux arch/amd64 UpstreamClient(Docker-Client/18.06.1-ce \\(darwin\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=5.865973ms http.response.status=404 http.response.written=157 instance.id=ddd2d3be-a3a2-4775-b468-131345a1e85d service=registry vars.digest="sha256:5aff3f5eea5bc757acfbd17cf27c4dd136890d4f82ef3541db410b222cefe8ec" vars.name="jz21/libreoffice/online" version=v2.6.2
10.2.42.0 - - [22/Sep/2018:08:11:44 +0000] "HEAD /v2/jz21/libreoffice/online/blobs/sha256:5aff3f5eea5bc757acfbd17cf27c4dd136890d4f82ef3541db410b222cefe8ec HTTP/1.1" 404 157 "" "docker/18.06.1-ce go/go1.10.3 git-commit/e68fc7a kernel/4.9.93-linuxkit-aufs os/linux arch/amd64 UpstreamClient(Docker-Client/18.06.1-ce \\(darwin\\))"
time="2018-09-22T08:11:44.960813306Z" level=error msg="response completed with error" auth.user.name=jz21 err.code="blob unknown" err.detail=sha256:f353a04f2566c0f9021d011c4996f4115d718092a4bb739154da20a611abb91d err.message="blob unknown to registry" go.version=go1.7.6 http.request.host=registry.XXXXXX.com http.request.id=2a116e6e-6388-4b8c-b9ea-7fda9c63b433 http.request.method=HEAD http.request.remoteaddr=172.16.164.118 http.request.uri="/v2/jz21/libreoffice/online/blobs/sha256:f353a04f2566c0f9021d011c4996f4115d718092a4bb739154da20a611abb91d" http.request.useragent="docker/18.06.1-ce go/go1.10.3 git-commit/e68fc7a kernel/4.9.93-linuxkit-aufs os/linux arch/amd64 UpstreamClient(Docker-Client/18.06.1-ce \\(darwin\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=4.496661ms http.response.status=404 http.response.written=157 instance.id=ddd2d3be-a3a2-4775-b468-131345a1e85d service=registry vars.digest="sha256:f353a04f2566c0f9021d011c4996f4115d718092a4bb739154da20a611abb91d" vars.name="jz21/libreoffice/online" version=v2.6.2
10.2.18.0 - - [22/Sep/2018:08:11:44 +0000] "HEAD /v2/jz21/libreoffice/online/blobs/sha256:f353a04f2566c0f9021d011c4996f4115d718092a4bb739154da20a611abb91d HTTP/1.1" 404 157 "" "docker/18.06.1-ce go/go1.10.3 git-commit/e68fc7a kernel/4.9.93-linuxkit-aufs os/linux arch/amd64 UpstreamClient(Docker-Client/18.06.1-ce \\(darwin\\))"
time="2018-09-22T08:11:44.962658059Z" level=error msg="response completed with error" auth.user.name=jz21 err.code="blob unknown" err.detail=sha256:704b3983e70c73c971cdacd8517454d433ce50b972520ca0c97f4423b9e82fe1 err.message="blob unknown to registry" go.version=go1.7.6 http.request.host=registry.XXXXXX.com http.request.id=ee9ba0b8-811a-44c6-99f6-e674542b2b91 http.request.method=HEAD http.request.remoteaddr=172.16.164.118 http.request.uri="/v2/jz21/libreoffice/online/blobs/sha256:704b3983e70c73c971cdacd8517454d433ce50b972520ca0c97f4423b9e82fe1" http.request.useragent="docker/18.06.1-ce go/go1.10.3 git-commit/e68fc7a kernel/4.9.93-linuxkit-aufs os/linux arch/amd64 UpstreamClient(Docker-Client/18.06.1-ce \\(darwin\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=5.590124ms http.response.status=404 http.response.written=157 instance.id=ddd2d3be-a3a2-4775-b468-131345a1e85d service=registry vars.digest="sha256:704b3983e70c73c971cdacd8517454d433ce50b972520ca0c97f4423b9e82fe1" vars.name="jz21/libreoffice/online" version=v2.6.2
10.2.6.0 - - [22/Sep/2018:08:11:44 +0000] "HEAD /v2/jz21/libreoffice/online/blobs/sha256:704b3983e70c73c971cdacd8517454d433ce50b972520ca0c97f4423b9e82fe1 HTTP/1.1" 404 157 "" "docker/18.06.1-ce go/go1.10.3 git-commit/e68fc7a kernel/4.9.93-linuxkit-aufs os/linux arch/amd64 UpstreamClient(Docker-Client/18.06.1-ce \\(darwin\\))"
10.2.18.0 - - [22/Sep/2018:08:11:45 +0000] "POST /v2/jz21/libreoffice/online/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/18.06.1-ce go/go1.10.3 git-commit/e68fc7a kernel/4.9.93-linuxkit-aufs os/linux arch/amd64 UpstreamClient(Docker-Client/18.06.1-ce \\(darwin\\))"
10.2.6.0 - - [22/Sep/2018:08:11:46 +0000] "POST /v2/jz21/libreoffice/online/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/18.06.1-ce go/go1.10.3 git-commit/e68fc7a kernel/4.9.93-linuxkit-aufs os/linux arch/amd64 UpstreamClient(Docker-Client/18.06.1-ce \\(darwin\\))"

If there is any other information you need, please let me know.

Thank you so much for your help! It means a lot to me!

Edited by Jude.X.Zhu