Unable to push to registry

Summary

We have setup gitlab on our AWS. And we are unable to push to registry.

Our Gitlab is on EC2 instance with ELB (because of HTTPS - TLS). There are snippets from gitlab omnibus configuration.

external_url 'https://gitlab.ourdomain.com'
# ...
gitlab_rails['registry_enabled'] = true
gitlab_rails['registry_host'] = "registry.ourdomain.com"
# ...
registry['enable'] = true
# ...
nginx['listen_port'] = 80
nginx['listen_https'] = false
# ...
nginx['real_ip_trusted_addresses'] = ['10.252.0.0/16']
nginx['real_ip_header'] = 'X-Forwarded-For'
nginx['real_ip_recursive'] = 'on'
# ...
registry_nginx['listen_port'] = 80
registry_nginx['listen_https'] = false
registry_nginx['proxy_set_headers'] = {
  "Host" => "$http_host",
  "X-Real-IP" => "$remote_addr",
  "X-Forwarded-For" => "$proxy_add_x_forwarded_for",
  "X-Forwarded-Proto" => "https",
  "X-Forwarded-Ssl" => "on"
}

All other options are default (commented). We use latest Gitlab (8.13.3). On server where i try to pull from is Docker 1.12.3 and Ubuntu 16.04.

Steps to reproduce

We use 2FA so for login to registry I use Access Token instead of password. User robot has role Developer in group docker. I'm able to pull from that registy but not push to it.

docker push registry.ourdomain.com/docker/chef

Expected behavior

Push image to registry

Actual behavior

500 error

Relevant logs and/or screenshots

DEBU[0011] Calling POST /v1.24/images/registry.ourdomain.com/docker/chef/push?tag=
DEBU[0011] hostDir: /etc/docker/certs.d/registry.ourdomain.com
DEBU[0011] hostDir: /etc/docker/certs.d/registry.ourdomain.com
DEBU[0011] Trying to push registry.ourdomain.com/docker/chef to https://registry.ourdomain.com v2
DEBU[0011] Pushing repository: registry.ourdomain.com/docker/chef:latest
DEBU[0011] Pushing layer: sha256:03a846941372adbcc90315ca234cd5d24d9e10206f32735d5eba100751a4347c
DEBU[0011] Pushing layer: sha256:c288ea0828c03d1b2cbf3fa6e24d1a2e7c19430aac94f9f6ec63aa5e82b71361
DEBU[0011] Pushing layer: sha256:011b303988d241a4ae28a6b82b0d8262751ef02910f0ae2265cb637504b72e36
DEBU[0012] Increasing token expiration to: 60 seconds
ERRO[0012] Upload failed, retrying: received unexpected HTTP status: 500 Internal Server Error
ERRO[0012] Upload failed, retrying: received unexpected HTTP status: 500 Internal Server Error
ERRO[0012] Upload failed, retrying: received unexpected HTTP status: 500 Internal Server Error

Output of checks

Results of GitLab application Check

Checking GitLab Shell ...

GitLab Shell version >= 3.6.6 ? ... OK (3.6.6)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
4/1 ... ok
5/2 ... ok
10/3 ... ok
14/4 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
Send ping to redis server: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Sidekiq ...

Running? ... yes
Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Reply by email ...

Reply by email is disabled in config/gitlab.yml

Checking Reply by email ... Finished

Checking LDAP ...

LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab ...

Git configured with autocrlf=input? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory setup correctly? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
projects have namespace: ...
4/1 ... yes
5/2 ... yes
10/3 ... yes
14/4 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.1.0 ? ... yes (2.3.1)
Your git bin path is "/opt/gitlab/embedded/bin/git"
Git version >= 2.7.3 ? ... yes (2.7.4)
Active users: 9

Checking GitLab ... Finished

Results of GitLab environment info

System information
System:         Ubuntu 16.04
Current User:   git
Using RVM:      no
Ruby Version:   2.3.1p112
Gem Version:    2.6.6
Bundler Version:1.13.5
Rake Version:   10.5.0
Sidekiq Version:4.2.1

GitLab information
Version:        8.13.3
Revision:       8d79ab3
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     postgresql
URL:            https://gitlab.ourdomain.com
HTTP Clone URL: https://gitlab.ourdomain.com/some-group/some-project.git
SSH Clone URL:  git@gitlab.ourdomain.com:some-group/some-project.git
Using LDAP:     no
Using Omniauth: no

GitLab Shell
Version:        3.6.6
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
Hooks:          /opt/gitlab/embedded/service/gitlab-shell/hooks/
Git:            /opt/gitlab/embedded/bin/git