docker push to container registry returns 500 error

Summary

Attempting to docker push any image to a private container registry fails to push any layers and instead loops retries until the command fails, resulting in a 500 Internal Service Error.

Steps to reproduce

  1. Enable container registry for a project on a private gitlab server.
  2. docker login to enable using the registry: docker login private.gitlab.url:5050 -u <username> -p <project-access-token>
  3. Acquire an image. This error occurred both for a private ~2GB image and the basic docker hello-world image, so use that: docker pull hello-world
  4. Tag the image so it can be pushed to the private container registry: docker tag <image-id> private.gitlab.url:5050/path/to/container/registry/hello-world
  5. Attempt to push the image: docker push private.gitlab.url:5050/path/to/container/registry/hello-world

What is the current bug behavior?

The push tries to push the layer(s), waiting a few seconds between each retry. It fails on all layers, and none ever show a progress bar. After a few retries, a 500 error is returned. Additionally, after the first push, the container registry gains an image that has nothing in it and gives the screen that says "This image has no active tags". Trying to run the push command again after the first time results in the same error, and the empty image in the container registry shows no changes.

The exact error is: received unexpected HTTP status: 500 Internal Server Error

What is the expected correct behavior?

The image should be pushed and available in the container registry.

Relevant logs and/or screenshots

Output of the docker push command:

% docker push private.gitlab.url:5050/path/to/container/registry/hello-world
Using default tag: latest
The push refers to repository [private.gitlab.url:5050/path/to/container/registry/hello-world]
a7866053acac: Retrying in 1 second
received unexpected HTTP status: 500 Internal Server Error

https://serverfault.com/questions/1000069/gitlab-docker-registry-reject-push-with-error-500

This is the only place online that I could find what appears to be the same error I'm facing, so it seems useful to post here. However, the solutions provided were attempted and did not fix the issue.

registry is already the owner of the files

drwxrwx---  2 registry git        4.0K Mar 27 15:35 registry

Results of GitLab environment info


```
System information
System:         Ubuntu 20.04
Proxy:          no
Current User:   git
Using RVM:      no
Ruby Version:   3.0.6p216
Gem Version:    3.4.18
Bundler Version:2.4.18
Rake Version:   13.0.6
Redis Version:  7.0.12
Sidekiq Version:6.5.7
Go Version:     unknown

GitLab information
Version:        16.3.2-ee
Revision:       fedb711c3d2
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     13.11
URL:            https://private.gitlab.url
HTTP Clone URL: https://private.gitlab.url/some-group/some-project.git
SSH Clone URL:  git@private.gitlab.url:some-group/some-project.git
Elasticsearch:  no
Geo:            no
Using LDAP:     yes
Using Omniauth: yes
Omniauth Providers:

GitLab Shell
Version:        14.26.0
Repository storages:
- default:      unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell

Gitaly
- default Address:      unix:/var/opt/gitlab/gitaly/gitaly.socket
- default Version:      16.3.2
- default Git Version:  2.41.0.gl1
```

Results of GitLab application Check

Expand for output related to the GitLab application check 


Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 14.26.0 ? ... OK (14.26.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... Server: ldapmain
not verifying SSL hostname of LDAPS server '<private.url>'
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
        User output sanitized. Found <number of users> users of 100 limit.

Checking LDAP ... Finished

Checking GitLab App ...

Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Cable config exists? ... yes
Resque config exists? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ...
note: i deleted these but they all say yes
Redis version >= 6.0.0? ... yes
Ruby version >= 3.0.6 ? ... yes (3.0.6)
Git user has default SSH configuration? ... yes
Active users: ... <number of active users>
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Elasticsearch version 7.x-8.x or OpenSearch version 1.x ... skipped (Advanced Search is disabled)
All migrations must be finished before doing a major upgrade ... skipped (Advanced Search is disabled)

Checking GitLab App ... Finished

Checking GitLab subtasks ... Finished

Possible fixes

https://serverfault.com/questions/1000069/gitlab-docker-registry-reject-push-with-error-500

This post is the closest thing to my issue that I've found online. Key similarities include every layer of the image failing, the exact way the layers retry uploading, and that the issue seems to be independent of the image size. This post recommends making sure the registry directory is owned by the registry user, but that was already the case on our gitlab instance.

We did find that the /docker/registry files mentioned in that post that are supposed to be at /var/opt/gitlab/gitlab-rails/shared/registry/docker/registry/ were missing. We tried manually creating them, but still encounter this error.

Edited by A. West