Code-Quality issue with Docker Hub rate limits
Summary
Running the code-quality job behind a corporate proxy fails when the codequality image tries to pull the codeclimate image due to the recent introduction of Docker Hub rate limits.
Steps to reproduce
- Clone/fork example project to your self-managed gitlab
- Enable runner for docker-in-docker ci execution support
- Check server IP address docker hub limits with that script
- Rerun code quality job for cloned/forked project
- Check docker limits again. That value will be decreased by 3.
Example Project
https://gitlab.com/AJIOB/codeclimate-limits
What is the current bug behavior?
Regularly code quality usage will kill all docker hub limits
Executing "step_script" stage of the job script
$ if ! docker info &>/dev/null; then # collapsed multi-line command
$ docker pull --quiet "$CODE_QUALITY_IMAGE"
registry.gitlab.com/gitlab-org/ci-cd/codequality:0.85.10-gitlab.1
$ docker run --env SOURCE_CODE="$PWD" --volume "$PWD":/code --volume /var/run/docker.sock:/var/run/docker.sock "$CODE_QUALITY_IMAGE" /code
Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
Unable to find image 'codeclimate/codeclimate:0.85.10' locally
docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.
See 'docker run --help'.
Could not install code climate engines for the repository at /code
What is the expected correct behavior?
Regularly code quality usage shouldn't dependent on docker hub limits
Relevant logs and/or screenshots
Before CI task rerunning:
After CI task rerunning:
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Ubuntu 16.04 Current User: git Using RVM: no Ruby Version: 2.7.2p137 Gem Version: 3.1.4 Bundler Version:2.1.4 Rake Version: 13.0.1 Redis Version: 5.0.9 Git Version: 2.29.0 Sidekiq Version:5.2.9 Go Version: unknown GitLab information Version: 13.7.1 Revision: c97c8073a0e Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 12.4 URL: https://gitlab.example.com HTTP Clone URL: https://gitlab.example.com/some-group/some-project.git SSH Clone URL: git@gitlab.example.com:some-group/some-project.git Using LDAP: no Using Omniauth: yes Omniauth Providers: github GitLab Shell Version: 13.14.0 Repository storage paths: - default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.14.0 ? ... OK (13.14.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 2/1 ... yes 5/4 ... yes 5/5 ... yes 7/6 ... yes 7/7 ... yes 7/14 ... yes 10/16 ... yes 10/17 ... yes 7/18 ... yes 9/19 ... yes 8/20 ... yes 10/21 ... yes 13/25 ... yes 3/26 ... yes 7/27 ... yes 16/28 ... yes 10/29 ... yes 16/31 ... yes 22/32 ... yes 7/33 ... yes 7/34 ... yes 18/35 ... yes 18/36 ... yes 18/37 ... yes 23/38 ... yes 24/39 ... yes 22/40 ... yes 3/41 ... yes 22/42 ... yes 24/43 ... yes 26/44 ... yes 13/45 ... yes 26/46 ... yes 26/47 ... yes 13/48 ... yes 13/54 ... yes 30/55 ... yes 30/56 ... yes 9/57 ... yes 31/58 ... yes 7/59 ... yes 33/60 ... yes 33/62 ... yes 33/63 ... yes 33/64 ... yes 33/65 ... yes 35/66 ... yes 35/67 ... yes 35/68 ... yes 40/69 ... yes 35/70 ... yes 37/71 ... yes 35/72 ... yes 39/74 ... yes 49/75 ... yes 51/77 ... yes 52/78 ... yes 40/79 ... yes 39/80 ... yes 39/81 ... yes 38/83 ... yes 5/84 ... yes 55/85 ... yes 39/87 ... yes 37/88 ... yes 39/89 ... yes 58/90 ... yes 5/91 ... yes 60/92 ... yes 49/93 ... yes 95/94 ... yes 49/95 ... yes 49/96 ... yes 49/97 ... yes 35/98 ... yes 38/99 ... yes 38/100 ... yes 38/101 ... yes 38/102 ... yes 38/103 ... yes 38/104 ... yes 38/105 ... yes 38/106 ... yes 38/107 ... yes 35/108 ... yes 38/110 ... yes 55/111 ... yes 55/112 ... yes 18/113 ... yes 98/114 ... yes 35/115 ... yes 18/116 ... yes 49/120 ... yes 9/121 ... yes 18/122 ... yes 37/123 ... yes 49/124 ... yes 35/125 ... yes 100/126 ... yes 100/127 ... yes 102/129 ... yes 100/130 ... yes 102/131 ... yes 100/132 ... yes 110/133 ... yes 102/134 ... yes 57/135 ... yes 109/137 ... yes 35/138 ... yes 109/139 ... yes 110/140 ... yes 113/141 ... yes 115/142 ... yes 110/143 ... yes 9/144 ... yes 117/145 ... yes 37/147 ... yes 49/148 ... yes 117/150 ... yes 120/151 ... yes 120/152 ... yes 120/153 ... yes 121/154 ... yes 120/155 ... yes 121/156 ... yes 121/157 ... yes 121/158 ... yes 122/159 ... yes 122/160 ... yes 120/161 ... yes 26/162 ... yes 122/163 ... yes 124/164 ... yes 5/165 ... yes 120/166 ... yes 26/167 ... yes 125/169 ... yes 122/170 ... yes 26/171 ... yes 119/173 ... yes 26/174 ... yes 126/175 ... yes 26/176 ... yes 26/177 ... yes 127/178 ... yes 127/179 ... yes 127/180 ... yes 122/181 ... yes 125/182 ... yes 129/183 ... yes 130/184 ... yes 132/185 ... yes 132/186 ... yes 129/187 ... yes 133/188 ... yes 139/189 ... yes 139/190 ... yes 26/191 ... yes 139/192 ... yes 144/193 ... yes 117/194 ... yes 133/195 ... yes 139/198 ... yes 139/200 ... yes 139/201 ... yes 146/202 ... yes 123/203 ... yes 139/204 ... yes 9/205 ... yes 123/206 ... yes 147/207 ... yes 139/208 ... yes 26/209 ... yes 139/210 ... yes 148/211 ... yes 148/212 ... yes 148/213 ... yes 148/214 ... yes 148/215 ... yes 148/216 ... yes 148/217 ... yes 148/218 ... yes 148/219 ... yes 152/220 ... yes 148/222 ... yes 148/224 ... yes 148/225 ... yes 148/226 ... yes 148/227 ... yes 148/228 ... yes 148/229 ... yes 153/230 ... yes 148/231 ... yes 153/232 ... yes 139/233 ... yes 148/235 ... yes 157/237 ... yes 139/238 ... yes 120/239 ... yes 120/240 ... yes 120/241 ... yes 38/242 ... yes 139/245 ... yes 137/248 ... yes 137/249 ... yes 120/250 ... yes 162/251 ... yes 162/252 ... yes 162/253 ... yes 162/254 ... yes 162/255 ... yes 162/256 ... yes 162/257 ... yes 162/258 ... yes 162/259 ... yes 162/260 ... yes 148/261 ... yes 118/262 ... yes 163/263 ... yes 162/264 ... yes 163/265 ... yes 163/266 ... yes 162/267 ... yes 162/268 ... yes 162/269 ... yes 169/270 ... yes 123/271 ... yes 148/272 ... yes 148/273 ... yes 148/274 ... yes 148/275 ... yes 148/276 ... yes 148/277 ... yes 9/278 ... yes 148/279 ... yes 148/280 ... yes 148/281 ... yes 148/282 ... yes 148/283 ... yes 148/284 ... yes 148/285 ... yes 148/286 ... yes 148/287 ... yes 148/288 ... yes 148/289 ... yes 148/290 ... yes 148/291 ... yes 148/292 ... yes 148/293 ... yes 170/294 ... yes 148/295 ... yes 148/296 ... yes 148/298 ... yes Redis version >= 4.0.0? ... yes Ruby version >= 2.7.2 ? ... yes (2.7.2) Git version >= 2.29.0 ? ... yes (2.29.0) Git user has default SSH configuration? ... yes Active users: ... 35 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
Possible fixes
In the Code-Quality template, pass the CODECLIMATE_IMAGE
and CODECLIMATE_VERSION
environment variables into the codequality docker container. Then we can point it to an internal repo/mirror.
Set image mirror
We would like to try out this approach to set the docker image mirror inside of our template, pointing it at the google mirror. It's still not known whether that would also use the mirror for the internal calls from the engines:install
step, but it does seem like it could be promising. Slack Link (Internal)
From #320815 (closed)
All other images seem to be pulled from registry.gitlab.com.
BTW the Dependency Proxy for Containers is currently not working for us. We are able to login to $CI_DEPENDENCY_PROXY_SERVER
but encounter 404 on pulling the image or simply get a JSON with unauthorized
. CI_DEPENDENCY_PROXY_*
is exposed to the jobs.
BTW if not mentioned before: we are a gold customer and foo and of course rely on that feature and some teams want to use it.