Docker Registry slow to load or will not load with larger amounts of images (S3 Storage)

Summary

Our Docker registry is fairly large because upon every push to our master branch we push a new docker image tagged with the given Git Sha (This allows us the opportunity to easily track what code is live at any given time as well as easily revery back, push hotfixes etc.) However this causes a bit of trouble if you want to take a look at the registry tab on the frontend.

It seems to get a 502 error every time we try to load the large repo - other repos load fine and without issue and the logs just seem to give 200 responses and 307's up until the second it times out. So I'm having a hard time figuring out what the exact problem is

Steps to reproduce

push a large number of images to the same registry with unique tags (we use S3 storage - so they arent hosted on our local instance) - as you get more images it will become slower and slower to load and eventually it will just stop loading altogether

Example Project

We're using Self-hosted EE and I dont have a ton of OSS images to push but if this is necessary I'm sure we can figure something out

(If you are using an older version of GitLab, this will also determine whether the bug has been fixed in a more recent version)

What is the current bug behavior?

The Registry page just fails to load with a 502 error but only on the repo with lots of images stored in S3

What is the expected correct behavior?

The Registry should load showing the given tags

Relevant logs and/or screenshots

(Paste any relevant logs - please use code blocks (```) to format console output, logs, and code as it's very hard to read otherwise.)

Output of checks

(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)

Results of GitLab environment info

Expand for output related to GitLab environment info

(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:env:info)
(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production)

Results of GitLab application Check

Checking GitLab Shell ...

GitLab Shell version >= 5.3.1 ? ... OK (5.3.1) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:root, or git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 2/1 ... ok 2/2 ... ok 2/4 ... ok 2/5 ... ok 2/6 ... ok 2/7 ... ok 2/8 ... ok 2/9 ... ok 2/10 ... ok 2/11 ... ok 2/12 ... ok 2/15 ... ok 2/17 ... ok 2/18 ... ok 2/23 ... ok 2/24 ... ok 2/29 ... ok 2/30 ... ok 13/31 ... ok 13/32 ... ok 13/33 ... ok 13/34 ... ok 13/35 ... ok 15/36 ... ok 2/37 ... ok 15/38 ... ok 13/39 ... ok 2/40 ... ok 2/41 ... ok 2/42 ... ok 2/43 ... ok 2/44 ... ok Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Access to /var/opt/gitlab/.ssh/authorized_keys: OK Send ping to redis server: OK gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Sidekiq ...

Running? ... yes Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Reply by email ...

Reply by email is disabled in config/gitlab.yml

Checking Reply by email ... Finished

Checking LDAP ...

LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab ...

Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... no Try fixing it: sudo chown -R git /var/opt/gitlab/gitlab-rails/uploads sudo find /var/opt/gitlab/gitlab-rails/uploads -type f -exec chmod 0644 {} ; sudo find /var/opt/gitlab/gitlab-rails/uploads -type d -not -path /var/opt/gitlab/gitlab-rails/uploads -exec chmod 0700 {} ; For more information see: doc/install/installation.md in section "GitLab" Please fix the error above and rerun the checks. Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 2/1 ... yes 2/2 ... yes 2/4 ... yes 2/5 ... yes 2/6 ... yes 2/7 ... yes 2/8 ... yes 2/9 ... yes 2/10 ... yes 2/11 ... yes 2/12 ... yes 2/15 ... yes 2/17 ... yes 2/18 ... yes 2/23 ... yes 2/24 ... yes 2/29 ... yes 2/30 ... yes 13/31 ... yes 13/32 ... yes 13/33 ... yes 13/34 ... yes 13/35 ... yes 15/36 ... yes 2/37 ... yes 15/38 ... yes 13/39 ... yes 2/40 ... yes 2/41 ... yes 2/42 ... yes 2/43 ... yes 2/44 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.3 ? ... yes (2.3.3) Git version >= 2.7.3 ? ... yes (2.13.4) Active users: ... 15 Elasticsearch version 5.1 - 5.3? ... skipped (elasticsearch is disabled)

Checking GitLab ... Finished

Possible fixes

My guess would be that pagination of registry pages would work (I'm assuming you're attempting to load all the images from S3 at once - I can check into the code and verify this if I have time when we get done with our current release at work