CI - dind no longer works with runners without DOCKER_TLS_CERTDIR: "" variable

Summary

As of last night, around 01:00 CET, all jobs using dind stopped working. (because docker:dind image has had a breaking change?) Similar issue for gitlab-com: gitlab-com/gl-infra/production#982 (closed) that one is obviously closed(because a workaround has been executed), but there should be one tracking it here as well to provide a long-term fix for us on-prem folks.

Steps to reproduce

• Set up a job using dind.
• ensure runner gets latest(stable?) image of docker:dind
• try using the docker daemon

Example Project

gitlab-com ran into the same issue today: gitlab-com/gl-infra/production#982 (closed)

What is the current bug behavior?

Cannot connect to the Docker daemon at tcp://docker:2375. Is the docker daemon running?

What is the expected correct behavior?

Docker commands should work.

Relevant logs and/or screenshots

Workaround is to configure a job variable like this:

variables:
  DOCKER_TLS_CERTDIR: ""

Output of checks

$ sudo gitlab-ctl status
run: alertmanager: (pid 33055) 26641s; run: log: (pid 1434) 695527s
run: crond: (pid 33089) 26640s; run: log: (pid 1527) 695526s
run: gitaly: (pid 33074) 26641s; run: log: (pid 1471) 695527s
run: gitlab-monitor: (pid 33106) 26640s; run: log: (pid 1482) 695527s
run: gitlab-workhorse: (pid 32942) 26643s; run: log: (pid 1448) 695527s
run: grafana: (pid 33118) 26640s; run: log: (pid 1499) 695527s
run: logrotate: (pid 187727) 1437s; run: log: (pid 1446) 695527s
run: nginx: (pid 33142) 26639s; run: log: (pid 1453) 695527s
run: node-exporter: (pid 33151) 26638s; run: log: (pid 1421) 695528s
run: postgres-exporter: (pid 33173) 26638s; run: log: (pid 1481) 695527s
run: postgresql: (pid 13697) 27596s; run: log: (pid 1470) 695527s
run: prometheus: (pid 33196) 26637s; run: log: (pid 1423) 695528s
run: redis: (pid 13723) 27595s; run: log: (pid 1420) 695528s
run: redis-exporter: (pid 33300) 26637s; run: log: (pid 1432) 695527s
run: sidekiq: (pid 33321) 26637s; run: log: (pid 1469) 695527s
run: unicorn: (pid 33336) 26636s; run: log: (pid 1417) 695528s

Results of GitLab environment info

Expand for output related to GitLab environment info

System information
System:		
Proxy:		no
Current User:	git
Using RVM:	no
Ruby Version:	2.6.3p62
Gem Version:	2.7.9
Bundler Version:1.17.3
Rake Version:	12.3.2
Redis Version:	3.2.12
Git Version:	2.21.0
Sidekiq Version:5.2.7
Go Version:	unknown

GitLab information
Version:	12.1.0-ee
Revision:	1f2e6f3f6d8
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	10.7
URL:		...
HTTP Clone URL:	...
SSH Clone URL:	...
Elasticsearch:	no
Geo:		no
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: crowd

GitLab Shell
Version:	9.3.0
Repository storage paths:
- default: 	/var/opt/gitlab/git-data/repositories
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell
Git:		/opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check

Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 9.3.0 ? ... OK (9.3.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK

Access to /var/opt/gitlab/.ssh/authorized_keys: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab App ...

Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ... 
3/1 ... yes
3/3 ... yes
9/4 ... yes
3/5 ... yes
44/6 ... yes
9/7 ... yes
47/8 ... yes
48/9 ... yes
50/10 ... yes
9/11 ... yes
61/12 ... yes
65/15 ... yes
9/16 ... yes
9/17 ... yes
61/18 ... yes
47/19 ... yes
470/20 ... yes
77/21 ... yes
77/22 ... yes
9/23 ... yes
17/25 ... yes
81/26 ... yes
83/29 ... yes
9/30 ... yes
9/31 ... yes
79/33 ... yes
61/35 ... yes
79/36 ... yes
9/37 ... yes
81/39 ... yes
23/40 ... yes
61/41 ... yes
106/42 ... yes
69/43 ... yes
81/44 ... yes
81/46 ... yes
9/47 ... yes
475/48 ... yes
81/49 ... yes
81/53 ... yes
81/55 ... yes
81/59 ... yes
81/60 ... yes
81/61 ... yes
81/63 ... yes
61/65 ... yes
119/66 ... yes
29/68 ... yes
119/69 ... yes
82/72 ... yes
124/78 ... yes
124/79 ... yes
124/80 ... yes
124/81 ... yes
124/82 ... yes
106/83 ... yes
79/86 ... yes
131/87 ... yes
9/88 ... yes
9/89 ... yes
185/90 ... yes
83/91 ... yes
83/92 ... yes
79/93 ... yes
81/94 ... yes
9/95 ... yes
9/96 ... yes
139/97 ... yes
96/98 ... yes
177/99 ... yes
182/100 ... yes
155/103 ... yes
185/104 ... yes
186/105 ... yes
186/106 ... yes
191/107 ... yes
191/108 ... yes
96/109 ... yes
96/110 ... yes
86/111 ... yes
470/120 ... yes
468/161 ... yes
9/246 ... yes
9/249 ... yes
9/250 ... yes
9/251 ... yes
9/252 ... yes
9/254 ... yes
9/256 ... yes
9/257 ... yes
9/258 ... yes
9/261 ... yes
9/262 ... yes
9/265 ... yes
9/268 ... yes
9/269 ... yes
9/270 ... yes
9/273 ... yes
9/274 ... yes
16/276 ... yes
161/277 ... yes
164/278 ... yes
155/280 ... yes
468/282 ... yes
155/283 ... yes
81/284 ... yes
210/285 ... yes
81/286 ... yes
9/287 ... yes
9/288 ... yes
164/289 ... yes
161/291 ... yes
164/292 ... yes
212/293 ... yes
212/294 ... yes
155/295 ... yes
155/296 ... yes
155/297 ... yes
141/298 ... yes
9/300 ... yes
472/301 ... yes
185/302 ... yes
23/303 ... yes
101/304 ... yes
217/306 ... yes
192/307 ... yes
192/308 ... yes
155/309 ... yes
16/310 ... yes
47/311 ... yes
225/312 ... yes
230/313 ... yes
186/314 ... yes
470/315 ... yes
200/316 ... yes
9/317 ... yes
81/318 ... yes
3/333 ... yes
61/334 ... yes
139/336 ... yes
9/337 ... yes
9/338 ... yes
81/339 ... yes
242/341 ... yes
139/342 ... yes
139/343 ... yes
9/344 ... yes
139/345 ... yes
81/346 ... yes
81/347 ... yes
81/348 ... yes
89/351 ... yes
81/352 ... yes
475/353 ... yes
239/356 ... yes
139/357 ... yes
197/358 ... yes
139/359 ... yes
139/360 ... yes
141/361 ... yes
139/362 ... yes
475/363 ... yes
139/364 ... yes
9/365 ... yes
139/366 ... yes
208/367 ... yes
61/368 ... yes
239/369 ... yes
263/370 ... yes
222/371 ... yes
265/373 ... yes
139/374 ... yes
262/375 ... yes
473/376 ... yes
473/377 ... yes
471/380 ... yes
186/383 ... yes
139/384 ... yes
270/385 ... yes
81/388 ... yes
280/389 ... yes
258/390 ... yes
2/391 ... yes
262/392 ... yes
375/393 ... yes
139/394 ... yes
270/395 ... yes
79/396 ... yes
376/397 ... yes
81/399 ... yes
295/400 ... yes
19/401 ... yes
295/404 ... yes
293/405 ... yes
295/406 ... yes
295/407 ... yes
473/408 ... yes
9/410 ... yes
81/411 ... yes
139/413 ... yes
83/414 ... yes
275/416 ... yes
275/417 ... yes
275/418 ... yes
275/419 ... yes
275/420 ... yes
275/421 ... yes
275/422 ... yes
79/423 ... yes
115/424 ... yes
83/425 ... yes
9/426 ... yes
106/427 ... yes
219/429 ... yes
231/430 ... yes
3/433 ... yes
239/434 ... yes
472/435 ... yes
24/436 ... yes
47/437 ... yes
275/439 ... yes
275/440 ... yes
275/441 ... yes
376/442 ... yes
24/443 ... yes
313/444 ... yes
287/445 ... yes
79/446 ... yes
79/447 ... yes
79/448 ... yes
275/449 ... yes
61/452 ... yes
182/453 ... yes
324/454 ... yes
141/455 ... yes
106/456 ... yes
83/457 ... yes
327/458 ... yes
327/459 ... yes
287/460 ... yes
236/461 ... yes
86/462 ... yes
280/464 ... yes
106/465 ... yes
376/466 ... yes
141/467 ... yes
336/468 ... yes
275/469 ... yes
231/470 ... yes
231/471 ... yes
219/472 ... yes
275/473 ... yes
375/474 ... yes
430/475 ... yes
9/476 ... yes
262/477 ... yes
115/483 ... yes
283/484 ... yes
195/485 ... yes
435/487 ... yes
231/489 ... yes
375/491 ... yes
61/492 ... yes
231/494 ... yes
24/495 ... yes
24/496 ... yes
377/499 ... yes
275/500 ... yes
45/501 ... yes
24/502 ... yes
470/503 ... yes
2/504 ... yes
192/505 ... yes
262/506 ... yes
378/507 ... yes
470/509 ... yes
262/511 ... yes
384/512 ... yes
320/514 ... yes
388/515 ... yes
275/516 ... yes
24/517 ... yes
81/519 ... yes
47/520 ... yes
376/521 ... yes
47/522 ... yes
101/527 ... yes
143/529 ... yes
42/530 ... yes
143/531 ... yes
405/532 ... yes
409/533 ... yes
411/534 ... yes
410/535 ... yes
403/536 ... yes
413/537 ... yes
412/538 ... yes
414/539 ... yes
415/540 ... yes
308/541 ... yes
143/542 ... yes
275/543 ... yes
421/544 ... yes
473/545 ... yes
101/546 ... yes
476/547 ... yes
9/549 ... yes
47/550 ... yes
47/551 ... yes
425/552 ... yes
47/553 ... yes
226/554 ... yes
425/555 ... yes
425/556 ... yes
239/557 ... yes
425/558 ... yes
426/559 ... yes
426/560 ... yes
425/561 ... yes
425/562 ... yes
425/563 ... yes
471/564 ... yes
426/565 ... yes
427/566 ... yes
425/567 ... yes
47/568 ... yes
81/569 ... yes
431/570 ... yes
430/571 ... yes
375/572 ... yes
375/573 ... yes
435/574 ... yes
47/575 ... yes
435/576 ... yes
47/577 ... yes
47/578 ... yes
47/579 ... yes
47/580 ... yes
47/581 ... yes
47/582 ... yes
47/583 ... yes
47/584 ... yes
307/585 ... yes
275/587 ... yes
435/588 ... yes
435/589 ... yes
106/591 ... yes
9/593 ... yes
471/596 ... yes
2/597 ... yes
115/598 ... yes
443/599 ... yes
275/600 ... yes
443/601 ... yes
443/602 ... yes
275/603 ... yes
185/604 ... yes
338/605 ... yes
275/606 ... yes
275/607 ... yes
375/608 ... yes
375/609 ... yes
205/610 ... yes
61/611 ... yes
475/612 ... yes
239/613 ... yes
459/614 ... yes
275/615 ... yes
430/616 ... yes
101/617 ... yes
275/618 ... yes
101/619 ... yes
435/620 ... yes
469/623 ... yes
469/624 ... yes
469/625 ... yes
469/626 ... yes
469/627 ... yes
469/628 ... yes
469/629 ... yes
469/630 ... yes
469/631 ... yes
469/632 ... yes
469/633 ... yes
469/634 ... yes
469/635 ... yes
469/636 ... yes
469/638 ... yes
469/639 ... yes
469/640 ... yes
469/641 ... yes
468/643 ... yes
469/644 ... yes
468/645 ... yes
75/646 ... yes
472/647 ... yes
471/648 ... yes
275/650 ... yes
470/652 ... yes
370/653 ... yes
275/655 ... yes
75/656 ... yes
471/657 ... yes
475/658 ... yes
470/659 ... yes
475/660 ... yes
473/661 ... yes
473/662 ... yes
470/663 ... yes
473/664 ... yes
470/666 ... yes
471/667 ... yes
75/668 ... yes
473/670 ... yes
473/671 ... yes
473/672 ... yes
470/673 ... yes
470/674 ... yes
471/675 ... yes
481/676 ... yes
482/677 ... yes
482/678 ... yes
482/679 ... yes
482/680 ... yes
476/681 ... yes
471/682 ... yes
492/684 ... yes
470/685 ... yes
275/687 ... yes
497/688 ... yes
492/689 ... yes
492/690 ... yes
155/691 ... yes
9/692 ... yes
500/693 ... yes
500/694 ... yes
101/695 ... yes
47/696 ... yes
471/697 ... yes
500/698 ... yes
503/699 ... yes
503/700 ... yes
503/701 ... yes
503/702 ... yes
503/703 ... yes
492/704 ... yes
503/705 ... yes
503/706 ... yes
512/707 ... yes
231/708 ... yes
500/711 ... yes
83/712 ... yes
513/715 ... yes
513/716 ... yes
515/717 ... yes
24/718 ... yes
239/719 ... yes
396/721 ... yes
376/722 ... yes
375/723 ... yes
83/724 ... yes
519/725 ... yes
503/726 ... yes
503/727 ... yes
9/728 ... yes
9/729 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.3)
Git version >= 2.21.0 ? ... yes (2.21.0)
Git user has default SSH configuration? ... yes
Active users: ... 430
Elasticsearch version 5.6 - 6.x? ... skipped (elasticsearch is disabled)

Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

Possible fixes

(If you can, link to the line of code that might be responsible for the problem)