Incorrect docker registry path results in 403 and ultimately rack_attack blocking
Summary
Using an incorrect registry path in my CI-job resulted in "access denied" rather than "no such registry". This makes it harder to debug and had the side-effect that my runner was blocked.
Steps to reproduce
In my .gitlab-ci.yml
I first did a docker login registry.mydomain.com
which succeeded.
I then built my image and tried pushing it with docker push registry.mydomain.com/thegroup/nonexistent-projectname:master
.
This resulted in output like:
$ docker push $CONTAINER_TEST_IMAGE
The push refers to a repository [registry.mydomain.com/thegroup/nonexistent-projectname]
ca3062dfa464: Preparing
851f3e348c69: Preparing
e27a10675c56: Preparing
d2ca1d857e2c: Waiting
06f4de5fefea: Waiting
851f3e348c69: Waiting
e27a10675c56: Waiting
denied: requested access to the resource is denied
Aka "access denied", rather than "that registry does not exist". In the end this also resulted in the runner getting blocked by gitlab.
What is the current bug behavior?
docker push
results in access denied.
What is the expected correct behavior?
docker push
should result in something similar to http 404.
Results of GitLab environment info
Expand for output related to GitLab environment info
$ sudo gitlab-rake gitlab:env:info
System information System: Ubuntu 16.04 Current User: git Using RVM: no Ruby Version: 2.3.6p384 Gem Version: 2.6.13 Bundler Version:1.13.7 Rake Version: 12.3.0 Redis Version: 3.2.11 Git Version: 2.14.3 Sidekiq Version:5.0.5 Go Version: unknown
GitLab information Version: 10.4.0 Revision: 128ef10 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql URL: https://gitlab.skarp.dk HTTP Clone URL: https://gitlab.skarp.dk/some-group/some-project.git SSH Clone URL: git@gitlab.skarp.dk:some-group/some-project.git Using LDAP: no Using Omniauth: no
GitLab Shell Version: 5.11.0 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
$ sudo gitlab-rake gitlab:check SANITIZE=true Checking GitLab Shell ...
GitLab Shell version >= 5.11.0 ? ... OK (5.11.0) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:root, or git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 4/1 ... ok 4/2 ... ok 5/3 ... ok 5/4 ... ok 6/5 ... ok 6/6 ... ok 5/7 ... ok 5/8 ... repository is empty 5/9 ... ok 5/11 ... ok 5/12 ... ok 5/13 ... ok 5/14 ... ok 3/15 ... ok 5/16 ... ok 5/17 ... ok 5/18 ... ok 5/19 ... ok 5/20 ... ok 3/21 ... ok 3/28 ... ok 5/29 ... ok 3/30 ... ok 7/31 ... ok 7/32 ... ok 3/33 ... ok 3/35 ... ok 3/36 ... ok 9/56 ... ok 9/57 ... ok 9/58 ... ok 9/59 ... ok 10/60 ... ok 3/71 ... ok 3/72 ... ok 13/74 ... repository is empty 13/75 ... ok 13/76 ... ok 14/77 ... ok 14/78 ... ok 14/79 ... ok 14/80 ... ok 14/81 ... ok 14/82 ... ok 15/83 ... ok 15/84 ... ok 15/85 ... ok 15/86 ... ok 15/87 ... ok 16/88 ... ok 5/89 ... ok 3/90 ... ok 17/91 ... ok 17/92 ... ok 18/93 ... ok 18/94 ... ok 18/95 ... ok 20/96 ... ok 20/97 ... ok 21/100 ... ok 31/101 ... ok 22/104 ... ok 23/106 ... ok 22/107 ... ok 25/108 ... ok 25/109 ... ok 25/110 ... ok 25/111 ... ok 22/112 ... ok 25/113 ... ok 25/114 ... ok 31/116 ... ok 31/117 ... ok 31/118 ... ok 31/119 ... ok 9/120 ... ok 9/121 ... ok 31/122 ... repository is empty 35/123 ... ok 35/124 ... ok 25/125 ... ok 25/126 ... ok 25/127 ... ok 25/128 ... ok 25/129 ... ok 25/130 ... ok 39/131 ... ok 22/133 ... ok 25/134 ... ok 27/135 ... ok 39/139 ... ok 9/140 ... ok 3/141 ... ok 42/142 ... ok 42/143 ... ok 20/144 ... ok 18/145 ... ok 3/146 ... ok 44/147 ... ok 39/148 ... ok 3/149 ... ok 46/151 ... ok 46/152 ... ok 9/153 ... ok 3/154 ... ok 53/155 ... ok 55/156 ... ok 21/157 ... ok 57/158 ... ok 57/159 ... ok 9/160 ... ok 58/161 ... ok 58/162 ... ok 3/163 ... ok 59/164 ... ok 59/165 ... ok 59/166 ... ok 61/168 ... ok 3/169 ... ok 3/170 ... ok 39/171 ... ok 39/172 ... ok 61/173 ... ok 3/174 ... ok 3/175 ... ok 62/176 ... ok 61/178 ... ok 61/179 ... ok 61/180 ... repository is empty 61/181 ... repository is empty 61/182 ... ok 61/183 ... repository is empty 9/184 ... ok 61/186 ... ok 39/187 ... ok 61/188 ... ok 61/189 ... ok 53/190 ... ok 3/191 ... ok 18/192 ... ok 66/193 ... ok 9/194 ... ok 66/195 ... ok 66/196 ... ok 66/197 ... ok 66/198 ... ok 67/199 ... ok 66/200 ... ok 66/201 ... ok 66/202 ... ok 21/203 ... ok 3/204 ... ok 66/205 ... ok 69/206 ... ok 66/207 ... ok 70/208 ... ok 70/209 ... ok 70/210 ... ok 70/211 ... ok 3/212 ... ok 9/213 ... ok 70/214 ... ok 18/215 ... ok 69/216 ... ok 69/217 ... ok 70/218 ... ok 69/219 ... ok 3/221 ... ok 9/222 ... repository is empty 9/223 ... ok 21/224 ... ok 67/226 ... repository is empty 62/228 ... ok 62/229 ... ok 62/230 ... ok 62/231 ... ok 62/232 ... ok 69/233 ... ok 21/234 ... ok 21/235 ... ok 9/236 ... ok 62/237 ... ok 21/238 ... ok 9/239 ... ok 3/240 ... ok 83/241 ... repository is empty 3/242 ... ok 62/243 ... ok 3/244 ... ok 84/245 ... ok 3/246 ... ok 3/247 ... ok 3/248 ... ok 39/249 ... ok 86/250 ... ok 39/251 ... ok 9/253 ... ok 86/254 ... ok 67/255 ... ok 3/256 ... ok 70/257 ... ok 9/258 ... ok 3/259 ... ok 67/260 ... ok 88/261 ... ok 66/262 ... ok 9/263 ... ok 92/264 ... ok 84/265 ... ok 67/266 ... ok 67/267 ... ok 3/268 ... ok 67/269 ... ok 67/270 ... ok 93/271 ... ok 93/272 ... ok 3/273 ... ok 67/274 ... ok 93/275 ... ok 93/276 ... ok 93/277 ... ok Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Reply by email is disabled in config/gitlab.yml Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... no Try fixing it: sudo chown -R git /var/opt/gitlab/gitlab-rails/uploads sudo find /var/opt/gitlab/gitlab-rails/uploads -type f -exec chmod 0644 {} ; sudo find /var/opt/gitlab/gitlab-rails/uploads -type d -not -path /var/opt/gitlab/gitlab-rails/uploads -exec chmod 0700 {} ; For more information see: doc/install/installation.md in section "GitLab" Please fix the error above and rerun the checks. Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 4/1 ... yes 4/2 ... yes 5/3 ... yes 5/4 ... yes 6/5 ... yes 6/6 ... yes 5/7 ... yes 5/8 ... yes 5/9 ... yes 5/11 ... yes 5/12 ... yes 5/13 ... yes 5/14 ... yes 3/15 ... yes 5/16 ... yes 5/17 ... yes 5/18 ... yes 5/19 ... yes 5/20 ... yes 3/21 ... yes 3/28 ... yes 5/29 ... yes 3/30 ... yes 7/31 ... yes 7/32 ... yes 3/33 ... yes 3/35 ... yes 3/36 ... yes 9/56 ... yes 9/57 ... yes 9/58 ... yes 9/59 ... yes 10/60 ... yes 3/71 ... yes 3/72 ... yes 13/74 ... yes 13/75 ... yes 13/76 ... yes 14/77 ... yes 14/78 ... yes 14/79 ... yes 14/80 ... yes 14/81 ... yes 14/82 ... yes 15/83 ... yes 15/84 ... yes 15/85 ... yes 15/86 ... yes 15/87 ... yes 16/88 ... yes 5/89 ... yes 3/90 ... yes 17/91 ... yes 17/92 ... yes 18/93 ... yes 18/94 ... yes 18/95 ... yes 20/96 ... yes 20/97 ... yes 21/100 ... yes 31/101 ... yes 22/104 ... yes 23/106 ... yes 22/107 ... yes 25/108 ... yes 25/109 ... yes 25/110 ... yes 25/111 ... yes 22/112 ... yes 25/113 ... yes 25/114 ... yes 31/116 ... yes 31/117 ... yes 31/118 ... yes 31/119 ... yes 9/120 ... yes 9/121 ... yes 31/122 ... yes 35/123 ... yes 35/124 ... yes 25/125 ... yes 25/126 ... yes 25/127 ... yes 25/128 ... yes 25/129 ... yes 25/130 ... yes 39/131 ... yes 22/133 ... yes 25/134 ... yes 27/135 ... yes 39/139 ... yes 9/140 ... yes 3/141 ... yes 42/142 ... yes 42/143 ... yes 20/144 ... yes 18/145 ... yes 3/146 ... yes 44/147 ... yes 39/148 ... yes 3/149 ... yes 46/151 ... yes 46/152 ... yes 9/153 ... yes 3/154 ... yes 53/155 ... yes 55/156 ... yes 21/157 ... yes 57/158 ... yes 57/159 ... yes 9/160 ... yes 58/161 ... yes 58/162 ... yes 3/163 ... yes 59/164 ... yes 59/165 ... yes 59/166 ... yes 61/168 ... yes 3/169 ... yes 3/170 ... yes 39/171 ... yes 39/172 ... yes 61/173 ... yes 3/174 ... yes 3/175 ... yes 62/176 ... yes 61/178 ... yes 61/179 ... yes 61/180 ... yes 61/181 ... yes 61/182 ... yes 61/183 ... yes 9/184 ... yes 61/186 ... yes 39/187 ... yes 61/188 ... yes 61/189 ... yes 53/190 ... yes 3/191 ... yes 18/192 ... yes 66/193 ... yes 9/194 ... yes 66/195 ... yes 66/196 ... yes 66/197 ... yes 66/198 ... yes 67/199 ... yes 66/200 ... yes 66/201 ... yes 66/202 ... yes 21/203 ... yes 3/204 ... yes 66/205 ... yes 69/206 ... yes 66/207 ... yes 70/208 ... yes 70/209 ... yes 70/210 ... yes 70/211 ... yes 3/212 ... yes 9/213 ... yes 70/214 ... yes 18/215 ... yes 69/216 ... yes 69/217 ... yes 70/218 ... yes 69/219 ... yes 3/221 ... yes 9/222 ... yes 9/223 ... yes 21/224 ... yes 67/226 ... yes 62/228 ... yes 62/229 ... yes 62/230 ... yes 62/231 ... yes 62/232 ... yes 69/233 ... yes 21/234 ... yes 21/235 ... yes 9/236 ... yes 62/237 ... yes 21/238 ... yes 9/239 ... yes 3/240 ... yes 83/241 ... yes 3/242 ... yes 62/243 ... yes 3/244 ... yes 84/245 ... yes 3/246 ... yes 3/247 ... yes 3/248 ... yes 39/249 ... yes 86/250 ... yes 39/251 ... yes 9/253 ... yes 86/254 ... yes 67/255 ... yes 3/256 ... yes 70/257 ... yes 9/258 ... yes 3/259 ... yes 67/260 ... yes 88/261 ... yes 66/262 ... yes 9/263 ... yes 92/264 ... yes 84/265 ... yes 67/266 ... yes 67/267 ... yes 3/268 ... yes 67/269 ... yes 67/270 ... yes 93/271 ... yes 93/272 ... yes 3/273 ... yes 67/274 ... yes 93/275 ... yes 93/276 ... yes 93/277 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.5 ? ... yes (2.3.6) Git version >= 2.7.3 ? ... yes (2.14.3) Git user has default SSH configuration? ... yes Active users: ... 29
Checking GitLab ... Finished