Code-Quality issue with Docker Hub rate limits

Summary

Running the code-quality job behind a corporate proxy fails when the codequality image tries to pull the codeclimate image due to the recent introduction of Docker Hub rate limits.

Steps to reproduce

• Clone/fork example project to your self-managed gitlab
• Enable runner for docker-in-docker ci execution support
• Check server IP address docker hub limits with that script
• Rerun code quality job for cloned/forked project
• Check docker limits again. That value will be decreased by 3.

Example Project

https://gitlab.com/AJIOB/codeclimate-limits

What is the current bug behavior?

Regularly code quality usage will kill all docker hub limits

Executing "step_script" stage of the job script
$ if ! docker info &>/dev/null; then # collapsed multi-line command
$ docker pull --quiet "$CODE_QUALITY_IMAGE"
registry.gitlab.com/gitlab-org/ci-cd/codequality:0.85.10-gitlab.1
$ docker run --env SOURCE_CODE="$PWD" --volume "$PWD":/code --volume /var/run/docker.sock:/var/run/docker.sock "$CODE_QUALITY_IMAGE" /code
Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
Unable to find image 'codeclimate/codeclimate:0.85.10' locally
docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.
See 'docker run --help'.
Could not install code climate engines for the repository at /code

What is the expected correct behavior?

Regularly code quality usage shouldn't dependent on docker hub limits

Relevant logs and/or screenshots

Before CI task rerunning:

After CI task rerunning:

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

System information
System:         Ubuntu 16.04
Current User:   git
Using RVM:      no
Ruby Version:   2.7.2p137
Gem Version:    3.1.4
Bundler Version:2.1.4
Rake Version:   13.0.1
Redis Version:  5.0.9
Git Version:    2.29.0
Sidekiq Version:5.2.9
Go Version:     unknown

GitLab information
Version:        13.7.1
Revision:       c97c8073a0e
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     12.4
URL:            https://gitlab.example.com
HTTP Clone URL: https://gitlab.example.com/some-group/some-project.git
SSH Clone URL:  git@gitlab.example.com:some-group/some-project.git
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers: github

GitLab Shell
Version:        13.14.0
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell
Git:            /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check

Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.14.0 ? ... OK (13.14.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
2/1 ... yes
5/4 ... yes
5/5 ... yes
7/6 ... yes
7/7 ... yes
7/14 ... yes
10/16 ... yes
10/17 ... yes
7/18 ... yes
9/19 ... yes
8/20 ... yes
10/21 ... yes
13/25 ... yes
3/26 ... yes
7/27 ... yes
16/28 ... yes
10/29 ... yes
16/31 ... yes
22/32 ... yes
7/33 ... yes
7/34 ... yes
18/35 ... yes
18/36 ... yes
18/37 ... yes
23/38 ... yes
24/39 ... yes
22/40 ... yes
3/41 ... yes
22/42 ... yes
24/43 ... yes
26/44 ... yes
13/45 ... yes
26/46 ... yes
26/47 ... yes
13/48 ... yes
13/54 ... yes
30/55 ... yes
30/56 ... yes
9/57 ... yes
31/58 ... yes
7/59 ... yes
33/60 ... yes
33/62 ... yes
33/63 ... yes
33/64 ... yes
33/65 ... yes
35/66 ... yes
35/67 ... yes
35/68 ... yes
40/69 ... yes
35/70 ... yes
37/71 ... yes
35/72 ... yes
39/74 ... yes
49/75 ... yes
51/77 ... yes
52/78 ... yes
40/79 ... yes
39/80 ... yes
39/81 ... yes
38/83 ... yes
5/84 ... yes
55/85 ... yes
39/87 ... yes
37/88 ... yes
39/89 ... yes
58/90 ... yes
5/91 ... yes
60/92 ... yes
49/93 ... yes
95/94 ... yes
49/95 ... yes
49/96 ... yes
49/97 ... yes
35/98 ... yes
38/99 ... yes
38/100 ... yes
38/101 ... yes
38/102 ... yes
38/103 ... yes
38/104 ... yes
38/105 ... yes
38/106 ... yes
38/107 ... yes
35/108 ... yes
38/110 ... yes
55/111 ... yes
55/112 ... yes
18/113 ... yes
98/114 ... yes
35/115 ... yes
18/116 ... yes
49/120 ... yes
9/121 ... yes
18/122 ... yes
37/123 ... yes
49/124 ... yes
35/125 ... yes
100/126 ... yes
100/127 ... yes
102/129 ... yes
100/130 ... yes
102/131 ... yes
100/132 ... yes
110/133 ... yes
102/134 ... yes
57/135 ... yes
109/137 ... yes
35/138 ... yes
109/139 ... yes
110/140 ... yes
113/141 ... yes
115/142 ... yes
110/143 ... yes
9/144 ... yes
117/145 ... yes
37/147 ... yes
49/148 ... yes
117/150 ... yes
120/151 ... yes
120/152 ... yes
120/153 ... yes
121/154 ... yes
120/155 ... yes
121/156 ... yes
121/157 ... yes
121/158 ... yes
122/159 ... yes
122/160 ... yes
120/161 ... yes
26/162 ... yes
122/163 ... yes
124/164 ... yes
5/165 ... yes
120/166 ... yes
26/167 ... yes
125/169 ... yes
122/170 ... yes
26/171 ... yes
119/173 ... yes
26/174 ... yes
126/175 ... yes
26/176 ... yes
26/177 ... yes
127/178 ... yes
127/179 ... yes
127/180 ... yes
122/181 ... yes
125/182 ... yes
129/183 ... yes
130/184 ... yes
132/185 ... yes
132/186 ... yes
129/187 ... yes
133/188 ... yes
139/189 ... yes
139/190 ... yes
26/191 ... yes
139/192 ... yes
144/193 ... yes
117/194 ... yes
133/195 ... yes
139/198 ... yes
139/200 ... yes
139/201 ... yes
146/202 ... yes
123/203 ... yes
139/204 ... yes
9/205 ... yes
123/206 ... yes
147/207 ... yes
139/208 ... yes
26/209 ... yes
139/210 ... yes
148/211 ... yes
148/212 ... yes
148/213 ... yes
148/214 ... yes
148/215 ... yes
148/216 ... yes
148/217 ... yes
148/218 ... yes
148/219 ... yes
152/220 ... yes
148/222 ... yes
148/224 ... yes
148/225 ... yes
148/226 ... yes
148/227 ... yes
148/228 ... yes
148/229 ... yes
153/230 ... yes
148/231 ... yes
153/232 ... yes
139/233 ... yes
148/235 ... yes
157/237 ... yes
139/238 ... yes
120/239 ... yes
120/240 ... yes
120/241 ... yes
38/242 ... yes
139/245 ... yes
137/248 ... yes
137/249 ... yes
120/250 ... yes
162/251 ... yes
162/252 ... yes
162/253 ... yes
162/254 ... yes
162/255 ... yes
162/256 ... yes
162/257 ... yes
162/258 ... yes
162/259 ... yes
162/260 ... yes
148/261 ... yes
118/262 ... yes
163/263 ... yes
162/264 ... yes
163/265 ... yes
163/266 ... yes
162/267 ... yes
162/268 ... yes
162/269 ... yes
169/270 ... yes
123/271 ... yes
148/272 ... yes
148/273 ... yes
148/274 ... yes
148/275 ... yes
148/276 ... yes
148/277 ... yes
9/278 ... yes
148/279 ... yes
148/280 ... yes
148/281 ... yes
148/282 ... yes
148/283 ... yes
148/284 ... yes
148/285 ... yes
148/286 ... yes
148/287 ... yes
148/288 ... yes
148/289 ... yes
148/290 ... yes
148/291 ... yes
148/292 ... yes
148/293 ... yes
170/294 ... yes
148/295 ... yes
148/296 ... yes
148/298 ... yes
Redis version >= 4.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.2)
Git version >= 2.29.0 ? ... yes (2.29.0)
Git user has default SSH configuration? ... yes
Active users: ... 35
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished

Possible fixes

In the Code-Quality template, pass the CODECLIMATE_IMAGE and CODECLIMATE_VERSION environment variables into the codequality docker container. Then we can point it to an internal repo/mirror.

Set image mirror

We would like to try out this approach to set the docker image mirror inside of our template, pointing it at the google mirror. It's still not known whether that would also use the mirror for the internal calls from the engines:install step, but it does seem like it could be promising. Slack Link (Internal)

From #320815 (closed)

All other images seem to be pulled from registry.gitlab.com.

BTW the Dependency Proxy for Containers is currently not working for us. We are able to login to $CI_DEPENDENCY_PROXY_SERVER but encounter 404 on pulling the image or simply get a JSON with unauthorized. CI_DEPENDENCY_PROXY_* is exposed to the jobs.

BTW if not mentioned before: we are a gold customer and foo and of course rely on that feature and some teams want to use it.