Cannot push to gitlab registry with docker 1.9.1

Summary

I tried to push a docker image using docker v1.9.1 to Gitlab CE v11.8.0 internal docker registry, which failed Received unexpected HTTP status: 500 Internal Server Error.

The reason is that:

Gitlab v11.8.0 vendors docker distribution v2.7.1
docker v1.9.1 pushes using manifest format v2 schema1 (v2 schema2 was introduced with docker v1.10)
docker distribution disabled format v2 schema1 by default in v2.7.0 (it can be enabled back in)

Steps to reproduce

boot a Gitlab CE v11.8.0 instance with registry feature enabled
create a project with docker registry enabled
try to push a docker image to the registry using docker 1.9.1 client (client versions >= 1.10 will push using schema2 by default).

What is the current bug behavior?

Push fails with error 500 on the Gitlab registry side.

root@staging4-ci:~# docker push localhost:5000/root/test-gitlab-registry/app-deployer:0.0.1pre0
The push refers to a repository [localhost:5000/root/test-gitlab-registry/app-deployer] (len: 1)
67fe31c59d4d: Image already exists
64d025828e78: Image already exists
f35ff56700b9: Image already exists
f9caebdd5438: Image already exists
c1c86cabc6e7: Image already exists
545d20ffa16f: Image already exists
f6c9b84f80e0: Image already exists
563a7a0534cc: Image already exists
2ee40e7f1315: Image already exists
1c379e84cde4: Image already exists
13ff8a56b7fc: Image already exists
8f70458ec74e: Image already exists
06e326421855: Pushed
0.0.1pre0: digest: sha256:9ec941e3478b7b27e2458e879597476ce5ed4a8eaf34b0e2b72a2f5992881cb3 size: 37390
Received unexpected HTTP status: 500 Internal Server Error

What is the expected correct behavior?

The docker push should succeed and the pushed image visible in the Gitlab project's Registry page.

Relevant logs

# gitlab-ctl tail registry
2019-02-25_10:34:50.09471 time="2019-02-25T10:34:50.094629721Z" level=error msg="response completed with error" auth.user.name=root err.code=unknown err.detail="manifest schema v1 unsupported" err.message="unknown error" go.version=go1.10.3 http.request.host="localhost:5000" http.request.id=fd3e5cfa-be1a-40f6-ace6-c8cce39f07b6 http.request.method=PUT http.request.remoteaddr="172.17.42.1:39580" http.request.uri="/v2/root/test-gitlab-registry/app-deployer/manifests/0.0.1pre0" http.request.useragent="docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/4.4.0-116-generic os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=6.287747ms http.response.status=500 http.response.written=70 vars.name="root/test-gitlab-registry/app-deployer" vars.reference=0.0.1pre0

Details of package version

Provide the package version installation details

root@gitlab:/# dpkg-query -l "gitlab-*"
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                           Version                      Architecture                 Description
+++-==============================================-============================-============================-=================================================================================================
ii  gitlab-ce                                      11.8.0-ce.0                  amd64                        GitLab Community Edition (including NGINX, Postgres, Redis)
un  gitlab-ee                                                                                    (no description available)

Environment details

Operating System: Ubuntu 16.04 LTS
Installation Target, remove incorrect values:
- Bare Metal Machine
Installation Type, remove incorrect values:
- New Installation
Is there any other software running on the machine: docker v1.9.1
Is this a single or multiple node installation?
- Single node (dockerized)
Resources
- CPU: Intel(R) Core(TM) i7-4700MQ CPU @ 2.40GHz
- Memory total: 16 Gb

Configuration details

Dump of ${GITLAB_OMNIBUS_CONFIG} on the gitlab container.

Provide the relevant sections of `/etc/gitlab/gitlab.rb`

# https://docs.gitlab.com/omnibus/settings/nginx.html#supporting-proxied-ssl
external_url 'http://gitlab.192.168.122.1.xip.io:8080'
registry_external_url 'http://192.168.122.1:5000'
registry['registry_http_addr'] = "0.0.0.0:9089"
nginx['listen_port'] = 80
#nginx['redirect_http_to_https'] = true
#nginx['ssl_certificate'] = '/certificates/allinone.crt'
#nginx['ssl_certificate_key'] = '/certificates/server.key'
#letsencrypt['enable'] = false
#gitlab_rails['gitlab_https'] = false
#gitlab_rails['gitlab_port'] = 443
# https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/docker/README.md#expose-gitlab-on-different-ports
gitlab_rails['gitlab_shell_ssh_port'] = 8022
prometheus_monitoring['enable'] = true
# enable access to prometheus UI
prometheus['listen_address'] = '0.0.0.0:9090'
prometheus['monitor_kubernetes'] = false
prometheus['scrape_configs'] = [{
  'job_name' => 'federate',
  'scrape_interval' => '15s',
  'honor_labels' => true,
  'metrics_path' => '/federate',
  'params' => {'match[]' => ['{__name__="up",app="prometheus"}']},
  'static_configs' => [{'targets' => ['192.168.33.220:30090']}]
}]
alertmanager['enable'] = true

enable access to alertmanager UI
alertmanager['listen_address'] = '0.0.0.0:9093'
alertmanager['admin_email'] = 'gitlab-admin@example.com'
alertmanager['receivers'] = [{'name' => 'custom-receiver','email_configs' => ['to' => 'gitlab-admin@example.com', 'require_tls' => false]}]
alertmanager['default_receiver'] = 'custom-receiver'

gitlab_rails email configuration is required to implicitly configure alertmanager somehow
gitlab_rails['gitlab_email_enabled'] = true
gitlab_rails['gitlab_email_from'] = 'gitlab@example.com'
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = '192.168.33.220'
gitlab_rails['smtp_port'] = '30025'
gitlab_rails['smtp_tls'] = false
gitlab_rails['smtp_openssl_verify_mode'] = 'none'
gitlab_rails['smtp_enable_starttls_auto'] = false
gitlab_rails['smtp_ssl'] = false
gitlab_rails['smtp_force_ssl'] = false