Cannot push to gitlab registry with docker 1.9.1
Summary
I tried to push a docker image using docker v1.9.1 to Gitlab CE v11.8.0 internal docker registry,
which failed Received unexpected HTTP status: 500 Internal Server Error
.
The reason is that:
- Gitlab v11.8.0 vendors docker distribution v2.7.1
- docker v1.9.1 pushes using manifest format v2 schema1 (v2 schema2 was introduced with docker v1.10)
- docker distribution disabled format v2 schema1 by default in v2.7.0 (it can be enabled back in)
Steps to reproduce
- boot a Gitlab CE v11.8.0 instance with registry feature enabled
- create a project with docker registry enabled
- try to push a docker image to the registry using docker 1.9.1 client (client versions >= 1.10 will push using schema2 by default).
What is the current bug behavior?
Push fails with error 500 on the Gitlab registry side.
root@staging4-ci:~# docker push localhost:5000/root/test-gitlab-registry/app-deployer:0.0.1pre0
The push refers to a repository [localhost:5000/root/test-gitlab-registry/app-deployer] (len: 1)
67fe31c59d4d: Image already exists
64d025828e78: Image already exists
f35ff56700b9: Image already exists
f9caebdd5438: Image already exists
c1c86cabc6e7: Image already exists
545d20ffa16f: Image already exists
f6c9b84f80e0: Image already exists
563a7a0534cc: Image already exists
2ee40e7f1315: Image already exists
1c379e84cde4: Image already exists
13ff8a56b7fc: Image already exists
8f70458ec74e: Image already exists
06e326421855: Pushed
0.0.1pre0: digest: sha256:9ec941e3478b7b27e2458e879597476ce5ed4a8eaf34b0e2b72a2f5992881cb3 size: 37390
Received unexpected HTTP status: 500 Internal Server Error
What is the expected correct behavior?
The docker push should succeed and the pushed image visible in the Gitlab project's Registry page.
Relevant logs
Relevant logs
# gitlab-ctl tail registry 2019-02-25_10:34:50.09471 time="2019-02-25T10:34:50.094629721Z" level=error msg="response completed with error" auth.user.name=root err.code=unknown err.detail="manifest schema v1 unsupported" err.message="unknown error" go.version=go1.10.3 http.request.host="localhost:5000" http.request.id=fd3e5cfa-be1a-40f6-ace6-c8cce39f07b6 http.request.method=PUT http.request.remoteaddr="172.17.42.1:39580" http.request.uri="/v2/root/test-gitlab-registry/app-deployer/manifests/0.0.1pre0" http.request.useragent="docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/4.4.0-116-generic os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=6.287747ms http.response.status=500 http.response.written=70 vars.name="root/test-gitlab-registry/app-deployer" vars.reference=0.0.1pre0
Details of package version
Provide the package version installation details
root@gitlab:/# dpkg-query -l "gitlab-*" Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-==============================================-============================-============================-================================================================================================= ii gitlab-ce 11.8.0-ce.0 amd64 GitLab Community Edition (including NGINX, Postgres, Redis) un gitlab-ee (no description available)
Environment details
- Operating System:
Ubuntu 16.04 LTS
- Installation Target, remove incorrect values:
- Bare Metal Machine
- Installation Type, remove incorrect values:
- New Installation
- Is there any other software running on the machine:
docker v1.9.1
- Is this a single or multiple node installation?
- Single node (dockerized)
- Resources
- CPU:
Intel(R) Core(TM) i7-4700MQ CPU @ 2.40GHz
- Memory total:
16 Gb
- CPU:
Configuration details
Dump of ${GITLAB_OMNIBUS_CONFIG}
on the gitlab container.
Provide the relevant sections of `/etc/gitlab/gitlab.rb`
# https://docs.gitlab.com/omnibus/settings/nginx.html#supporting-proxied-ssl external_url 'http://gitlab.192.168.122.1.xip.io:8080' registry_external_url 'http://192.168.122.1:5000' registry['registry_http_addr'] = "0.0.0.0:9089" nginx['listen_port'] = 80 #nginx['redirect_http_to_https'] = true #nginx['ssl_certificate'] = '/certificates/allinone.crt' #nginx['ssl_certificate_key'] = '/certificates/server.key' #letsencrypt['enable'] = false #gitlab_rails['gitlab_https'] = false #gitlab_rails['gitlab_port'] = 443 # https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/docker/README.md#expose-gitlab-on-different-ports gitlab_rails['gitlab_shell_ssh_port'] = 8022 prometheus_monitoring['enable'] = true # enable access to prometheus UI prometheus['listen_address'] = '0.0.0.0:9090' prometheus['monitor_kubernetes'] = false prometheus['scrape_configs'] = [{ 'job_name' => 'federate', 'scrape_interval' => '15s', 'honor_labels' => true, 'metrics_path' => '/federate', 'params' => {'match[]' => ['{__name__="up",app="prometheus"}']}, 'static_configs' => [{'targets' => ['192.168.33.220:30090']}] }]alertmanager['enable'] = true
enable access to alertmanager UI
alertmanager['listen_address'] = '0.0.0.0:9093' alertmanager['admin_email'] = 'gitlab-admin@example.com' alertmanager['receivers'] = [{'name' => 'custom-receiver','email_configs' => ['to' => 'gitlab-admin@example.com', 'require_tls' => false]}] alertmanager['default_receiver'] = 'custom-receiver'
gitlab_rails email configuration is required to implicitly configure alertmanager somehow
gitlab_rails['gitlab_email_enabled'] = true gitlab_rails['gitlab_email_from'] = 'gitlab@example.com' gitlab_rails['smtp_enable'] = true gitlab_rails['smtp_address'] = '192.168.33.220' gitlab_rails['smtp_port'] = '30025' gitlab_rails['smtp_tls'] = false gitlab_rails['smtp_openssl_verify_mode'] = 'none' gitlab_rails['smtp_enable_starttls_auto'] = false gitlab_rails['smtp_ssl'] = false gitlab_rails['smtp_force_ssl'] = false