Container Registry refuses connections after moving data to other location
Summary
Container Registry stops working and refuses all connections after moving all registry files to other location.
Steps to reproduce
- Enable Container Registry for instance/project.
- Push any container to any project's Container Registry.
- Validate in GUI that the image has been uploaded.
- Change the
gitlab_rails['registry_path']
from default to other location. - Reconfigure and Restart GitLab instance.
- Try
docker login
from other computer - fails with either 502 Bad Gateway or 503 Service Unavailable. - Go to respective Container Registry via GUI - any previously uploaded tags will be missing.
What is the current bug behavior?
Our selfhosted GitLab instance was running out of the disk space, so we had decided to move multiple big folders and files to other location. One part of the data migration was also to move the Container Registry data. We have changed the gitlab_rails['registry_path']
as it is docummented here. Since then the problems started.
It has begun with refusal of all connections (especially docker login
) and giving us error 502 Bad Gateway. This has been accompanied with error log for registry in nginx/gitlab_registry_error.log
.
[error] 1862#0: *388276 connect() failed (111: Connection refused) while connecting to upstream, client: [REDACTED], server: [REDACTED], request: "GET /v2/ HTTP/1.1", upstream: "http://[::1]:5000/v2/", host: "[REDACTED]:5050"
I also found out that the GUI is also not showing any tags for the existing container images. Opening the container details just shown that currently there are no tags.
This error led me to #247510 GitLab issue. I had tried out the workaround with explicitly setting the registry['registry_http_addr']
to IPv4 address 127.0.0.1:5000
instead the default localhost:5000
. But this did not help at all and since then any connections to Container Registry failed with 503 Service Unavailable and no error logs being added to the nginx/gitlab_registry_error.log
. Reverting the registry['registry_http_addr']
setting back to default at least restored the error logs being added but connections always failed not with 502 Bad Gateway but with 503 Service Unavailable.
I have continued the investigation and found this forum thread talking about the Container Registry health checking. Disabling the registry['storagedriver_health_enabled']
indeed helped me and the connection was once again possible. But the data in GUI was still missing. Well but the health check is storage related right? So I have reverted the data migration for Container Registry and changed the gitlab_rails['registry_path']
back to default and the Container Registry functionality is back to normal with all existing data being available again.
To be clear, we have validated that the new location path configured is correct and that the file/folder permissions are the same as they were on the original location.
What is the expected correct behavior?
I would expect the changing of gitlab_rails['registry_path']
shall work without any issues. Or if there is something more to be done, I would expect it to be docummented in the relevant section.
Output of checks
This is selfhosted instance.
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Current User: git Using RVM: no Ruby Version: 2.7.5p203 Gem Version: 3.1.4 Bundler Version:2.1.4 Rake Version: 13.0.6 Redis Version: 6.0.16 Git Version: 2.33.1. Sidekiq Version:6.3.1 Go Version: unknown GitLab information Version: 14.7.0 Revision: abbf44bd6cf Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 12.7 URL: https://[REDACTED] HTTP Clone URL: https://[REDACTED]/some-group/some-project.git SSH Clone URL: git@[REDACTED]:some-group/some-project.git Using LDAP: yes Using Omniauth: yes Omniauth Providers: GitLab Shell Version: 13.22.2 Repository storage paths: - default: /data/git/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ...Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.22.2 ? ... OK (13.22.2) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes (cluster/worker) ... 1/1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... Server: ldapmain LDAP authentication... Success LDAP users with access to your GitLab server (only showing the first 100 results) User output sanitized. Found 100 users of 100 limit.
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units) Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units) Projects have namespace: ... 1/1 ... yes 17/4 ... yes 17/7 ... yes 17/8 ... yes 59/9 ... yes 30/10 ... yes 32/11 ... yes 33/12 ... yes 36/13 ... yes 41/14 ... yes 43/19 ... yes 44/20 ... yes 45/21 ... yes 46/22 ... yes 47/23 ... yes 48/24 ... yes 23/25 ... yes 53/26 ... yes 56/27 ... yes 17/28 ... yes 91/29 ... yes 20/30 ... yes 51/31 ... yes 58/33 ... yes 20/34 ... yes 59/35 ... yes 32/36 ... yes 64/37 ... yes 66/38 ... yes 27/40 ... yes 17/42 ... yes 69/46 ... yes 68/47 ... yes 68/48 ... yes 68/49 ... yes 68/50 ... yes 68/51 ... yes 24/53 ... yes 68/54 ... yes 26/55 ... yes 26/56 ... yes 24/57 ... yes 27/58 ... yes 27/59 ... yes 27/60 ... yes 74/62 ... yes 75/64 ... yes 78/65 ... yes 78/67 ... yes 25/68 ... yes 79/69 ... yes 79/70 ... yes 81/71 ... yes 81/72 ... yes 58/74 ... yes 83/75 ... yes 27/76 ... yes 84/77 ... yes 83/78 ... yes 27/79 ... yes 86/80 ... yes 91/82 ... yes 92/84 ... yes 92/85 ... yes 88/86 ... yes 91/87 ... yes 59/88 ... yes 67/89 ... yes 75/90 ... yes 95/91 ... yes 69/92 ... yes 27/93 ... yes 59/95 ... yes 28/96 ... yes 28/97 ... yes 28/98 ... yes 28/99 ... yes 26/101 ... yes 24/102 ... yes 101/103 ... yes 104/106 ... yes 104/107 ... yes 104/108 ... yes 104/109 ... yes 104/110 ... yes 104/112 ... yes 104/113 ... yes 104/115 ... yes 104/116 ... yes 104/117 ... yes 104/119 ... yes 104/120 ... yes 104/122 ... yes 104/124 ... yes 104/125 ... yes 104/126 ... yes 104/127 ... yes 104/128 ... yes 104/129 ... yes 104/130 ... yes 104/133 ... yes 104/134 ... yes 104/135 ... yes 104/137 ... yes 104/138 ... yes 104/139 ... yes 104/140 ... yes 104/141 ... yes 105/142 ... yes 105/143 ... yes 105/144 ... yes 105/145 ... yes 105/146 ... yes 104/147 ... yes 104/148 ... yes 104/149 ... yes 104/150 ... yes 104/151 ... yes 104/152 ... yes 104/153 ... yes 104/154 ... yes 104/155 ... yes 104/156 ... yes 104/157 ... yes 104/158 ... yes 104/159 ... yes 47/160 ... yes 104/161 ... yes 91/163 ... yes 48/165 ... yes 59/166 ... yes 86/167 ... yes 86/168 ... yes 109/169 ... yes 110/171 ... yes 31/172 ... yes 110/173 ... yes 112/174 ... yes 24/175 ... yes 56/176 ... yes 119/177 ... yes 110/178 ... yes 30/179 ... yes 127/181 ... yes 128/182 ... yes 129/183 ... yes 129/184 ... yes 130/185 ... yes 131/186 ... yes 132/187 ... yes 131/188 ... yes 133/189 ... yes 133/190 ... yes 134/191 ... yes 135/192 ... yes 135/193 ... yes 135/194 ... yes 135/195 ... yes 135/196 ... yes 135/197 ... yes 135/198 ... yes 135/199 ... yes 135/200 ... yes 135/201 ... yes 135/202 ... yes 136/203 ... yes 137/204 ... yes 137/205 ... yes 137/206 ... yes 137/207 ... yes 138/208 ... yes 138/209 ... yes 138/210 ... yes 138/211 ... yes 139/212 ... yes 139/213 ... yes 139/214 ... yes 139/215 ... yes 139/216 ... yes 139/217 ... yes 139/218 ... yes 139/219 ... yes 140/220 ... yes 140/221 ... yes 140/222 ... yes 140/223 ... yes 141/224 ... yes 141/225 ... yes 141/226 ... yes 142/227 ... yes 142/228 ... yes 142/229 ... yes 142/230 ... yes 142/231 ... yes 142/232 ... yes 142/233 ... yes 142/234 ... yes 142/235 ... yes 142/236 ... yes 142/237 ... yes 142/238 ... yes 143/239 ... yes 143/240 ... yes 143/241 ... yes 143/242 ... yes 143/243 ... yes 144/244 ... yes 144/245 ... yes 144/246 ... yes 144/247 ... yes 144/248 ... yes 144/249 ... yes 144/250 ... yes 145/251 ... yes 145/252 ... yes 146/253 ... yes 146/254 ... yes 146/255 ... yes 146/256 ... yes 146/257 ... yes 146/258 ... yes 146/259 ... yes 146/260 ... yes 146/261 ... yes 146/262 ... yes 146/263 ... yes 147/264 ... yes 147/265 ... yes 147/266 ... yes 147/267 ... yes 147/268 ... yes 147/269 ... yes 147/270 ... yes 147/271 ... yes 148/272 ... yes 148/273 ... yes 149/274 ... yes 149/275 ... yes 150/276 ... yes 151/277 ... yes 152/278 ... yes 153/279 ... yes 155/280 ... yes 156/281 ... yes 156/282 ... yes 156/283 ... yes 156/284 ... yes 157/285 ... yes 161/286 ... yes 159/287 ... yes 159/288 ... yes 159/289 ... yes 159/290 ... yes 159/291 ... yes 159/292 ... yes 159/293 ... yes 159/294 ... yes 159/295 ... yes 159/296 ... yes 159/297 ... yes 160/298 ... yes 165/299 ... yes 167/300 ... yes 168/302 ... yes 168/303 ... yes 168/304 ... yes 168/305 ... yes 168/306 ... yes 168/307 ... yes 168/308 ... yes 173/309 ... yes 24/310 ... yes 123/311 ... yes 179/312 ... yes 173/315 ... yes 48/316 ... yes 182/317 ... yes 26/318 ... yes 83/319 ... yes 83/320 ... yes 183/321 ... yes 183/322 ... yes 184/323 ... yes 83/324 ... yes 83/325 ... yes 184/326 ... yes 185/327 ... yes 100/328 ... yes 31/329 ... yes 29/330 ... yes 56/331 ... yes Redis version >= 5.0.0? ... yes Ruby version >= 2.7.2 ? ... yes (2.7.5) Git version >= 2.33.0 ? ... yes (2.33.1) Git user has default SSH configuration? ... yes Active users: ... 59 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished