CI - dind no longer works with runners without DOCKER_TLS_CERTDIR: "" variable
Summary
As of last night, around 01:00 CET, all jobs using dind stopped working. (because docker:dind image has had a breaking change?) Similar issue for gitlab-com: gitlab-com/gl-infra/production#982 (closed) that one is obviously closed(because a workaround has been executed), but there should be one tracking it here as well to provide a long-term fix for us on-prem folks.
Steps to reproduce
- Set up a job using dind.
- ensure runner gets latest(stable?) image of docker:dind
- try using the docker daemon
Example Project
gitlab-com ran into the same issue today: gitlab-com/gl-infra/production#982 (closed)
What is the current bug behavior?
Cannot connect to the Docker daemon at tcp://docker:2375. Is the docker daemon running?
What is the expected correct behavior?
Docker commands should work.
Relevant logs and/or screenshots
Workaround is to configure a job variable like this:
variables:
DOCKER_TLS_CERTDIR: ""
Output of checks
$ sudo gitlab-ctl status
run: alertmanager: (pid 33055) 26641s; run: log: (pid 1434) 695527s
run: crond: (pid 33089) 26640s; run: log: (pid 1527) 695526s
run: gitaly: (pid 33074) 26641s; run: log: (pid 1471) 695527s
run: gitlab-monitor: (pid 33106) 26640s; run: log: (pid 1482) 695527s
run: gitlab-workhorse: (pid 32942) 26643s; run: log: (pid 1448) 695527s
run: grafana: (pid 33118) 26640s; run: log: (pid 1499) 695527s
run: logrotate: (pid 187727) 1437s; run: log: (pid 1446) 695527s
run: nginx: (pid 33142) 26639s; run: log: (pid 1453) 695527s
run: node-exporter: (pid 33151) 26638s; run: log: (pid 1421) 695528s
run: postgres-exporter: (pid 33173) 26638s; run: log: (pid 1481) 695527s
run: postgresql: (pid 13697) 27596s; run: log: (pid 1470) 695527s
run: prometheus: (pid 33196) 26637s; run: log: (pid 1423) 695528s
run: redis: (pid 13723) 27595s; run: log: (pid 1420) 695528s
run: redis-exporter: (pid 33300) 26637s; run: log: (pid 1432) 695527s
run: sidekiq: (pid 33321) 26637s; run: log: (pid 1469) 695527s
run: unicorn: (pid 33336) 26636s; run: log: (pid 1417) 695528s
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Proxy: no Current User: git Using RVM: no Ruby Version: 2.6.3p62 Gem Version: 2.7.9 Bundler Version:1.17.3 Rake Version: 12.3.2 Redis Version: 3.2.12 Git Version: 2.21.0 Sidekiq Version:5.2.7 Go Version: unknown GitLab information Version: 12.1.0-ee Revision: 1f2e6f3f6d8 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 10.7 URL: ... HTTP Clone URL: ... SSH Clone URL: ... Elasticsearch: no Geo: no Using LDAP: no Using Omniauth: yes Omniauth Providers: crowd GitLab Shell Version: 9.3.0 Repository storage paths: - default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ... Checking GitLab Shell ... GitLab Shell: ... GitLab Shell version >= 9.3.0 ? ... OK (9.3.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK Access to /var/opt/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful Checking GitLab Shell ... Finished Checking Gitaly ... Gitaly: ... default ... OK Checking Gitaly ... Finished Checking Sidekiq ... Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1 Checking Sidekiq ... Finished Checking Incoming Email ... Incoming Email: ... Reply by email is disabled in config/gitlab.yml Checking Incoming Email ... Finished Checking LDAP ... LDAP: ... LDAP is disabled in config/gitlab.yml Checking LDAP ... Finished Checking GitLab App ... Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 3/1 ... yes 3/3 ... yes 9/4 ... yes 3/5 ... yes 44/6 ... yes 9/7 ... yes 47/8 ... yes 48/9 ... yes 50/10 ... yes 9/11 ... yes 61/12 ... yes 65/15 ... yes 9/16 ... yes 9/17 ... yes 61/18 ... yes 47/19 ... yes 470/20 ... yes 77/21 ... yes 77/22 ... yes 9/23 ... yes 17/25 ... yes 81/26 ... yes 83/29 ... yes 9/30 ... yes 9/31 ... yes 79/33 ... yes 61/35 ... yes 79/36 ... yes 9/37 ... yes 81/39 ... yes 23/40 ... yes 61/41 ... yes 106/42 ... yes 69/43 ... yes 81/44 ... yes 81/46 ... yes 9/47 ... yes 475/48 ... yes 81/49 ... yes 81/53 ... yes 81/55 ... yes 81/59 ... yes 81/60 ... yes 81/61 ... yes 81/63 ... yes 61/65 ... yes 119/66 ... yes 29/68 ... yes 119/69 ... yes 82/72 ... yes 124/78 ... yes 124/79 ... yes 124/80 ... yes 124/81 ... yes 124/82 ... yes 106/83 ... yes 79/86 ... yes 131/87 ... yes 9/88 ... yes 9/89 ... yes 185/90 ... yes 83/91 ... yes 83/92 ... yes 79/93 ... yes 81/94 ... yes 9/95 ... yes 9/96 ... yes 139/97 ... yes 96/98 ... yes 177/99 ... yes 182/100 ... yes 155/103 ... yes 185/104 ... yes 186/105 ... yes 186/106 ... yes 191/107 ... yes 191/108 ... yes 96/109 ... yes 96/110 ... yes 86/111 ... yes 470/120 ... yes 468/161 ... yes 9/246 ... yes 9/249 ... yes 9/250 ... yes 9/251 ... yes 9/252 ... yes 9/254 ... yes 9/256 ... yes 9/257 ... yes 9/258 ... yes 9/261 ... yes 9/262 ... yes 9/265 ... yes 9/268 ... yes 9/269 ... yes 9/270 ... yes 9/273 ... yes 9/274 ... yes 16/276 ... yes 161/277 ... yes 164/278 ... yes 155/280 ... yes 468/282 ... yes 155/283 ... yes 81/284 ... yes 210/285 ... yes 81/286 ... yes 9/287 ... yes 9/288 ... yes 164/289 ... yes 161/291 ... yes 164/292 ... yes 212/293 ... yes 212/294 ... yes 155/295 ... yes 155/296 ... yes 155/297 ... yes 141/298 ... yes 9/300 ... yes 472/301 ... yes 185/302 ... yes 23/303 ... yes 101/304 ... yes 217/306 ... yes 192/307 ... yes 192/308 ... yes 155/309 ... yes 16/310 ... yes 47/311 ... yes 225/312 ... yes 230/313 ... yes 186/314 ... yes 470/315 ... yes 200/316 ... yes 9/317 ... yes 81/318 ... yes 3/333 ... yes 61/334 ... yes 139/336 ... yes 9/337 ... yes 9/338 ... yes 81/339 ... yes 242/341 ... yes 139/342 ... yes 139/343 ... yes 9/344 ... yes 139/345 ... yes 81/346 ... yes 81/347 ... yes 81/348 ... yes 89/351 ... yes 81/352 ... yes 475/353 ... yes 239/356 ... yes 139/357 ... yes 197/358 ... yes 139/359 ... yes 139/360 ... yes 141/361 ... yes 139/362 ... yes 475/363 ... yes 139/364 ... yes 9/365 ... yes 139/366 ... yes 208/367 ... yes 61/368 ... yes 239/369 ... yes 263/370 ... yes 222/371 ... yes 265/373 ... yes 139/374 ... yes 262/375 ... yes 473/376 ... yes 473/377 ... yes 471/380 ... yes 186/383 ... yes 139/384 ... yes 270/385 ... yes 81/388 ... yes 280/389 ... yes 258/390 ... yes 2/391 ... yes 262/392 ... yes 375/393 ... yes 139/394 ... yes 270/395 ... yes 79/396 ... yes 376/397 ... yes 81/399 ... yes 295/400 ... yes 19/401 ... yes 295/404 ... yes 293/405 ... yes 295/406 ... yes 295/407 ... yes 473/408 ... yes 9/410 ... yes 81/411 ... yes 139/413 ... yes 83/414 ... yes 275/416 ... yes 275/417 ... yes 275/418 ... yes 275/419 ... yes 275/420 ... yes 275/421 ... yes 275/422 ... yes 79/423 ... yes 115/424 ... yes 83/425 ... yes 9/426 ... yes 106/427 ... yes 219/429 ... yes 231/430 ... yes 3/433 ... yes 239/434 ... yes 472/435 ... yes 24/436 ... yes 47/437 ... yes 275/439 ... yes 275/440 ... yes 275/441 ... yes 376/442 ... yes 24/443 ... yes 313/444 ... yes 287/445 ... yes 79/446 ... yes 79/447 ... yes 79/448 ... yes 275/449 ... yes 61/452 ... yes 182/453 ... yes 324/454 ... yes 141/455 ... yes 106/456 ... yes 83/457 ... yes 327/458 ... yes 327/459 ... yes 287/460 ... yes 236/461 ... yes 86/462 ... yes 280/464 ... yes 106/465 ... yes 376/466 ... yes 141/467 ... yes 336/468 ... yes 275/469 ... yes 231/470 ... yes 231/471 ... yes 219/472 ... yes 275/473 ... yes 375/474 ... yes 430/475 ... yes 9/476 ... yes 262/477 ... yes 115/483 ... yes 283/484 ... yes 195/485 ... yes 435/487 ... yes 231/489 ... yes 375/491 ... yes 61/492 ... yes 231/494 ... yes 24/495 ... yes 24/496 ... yes 377/499 ... yes 275/500 ... yes 45/501 ... yes 24/502 ... yes 470/503 ... yes 2/504 ... yes 192/505 ... yes 262/506 ... yes 378/507 ... yes 470/509 ... yes 262/511 ... yes 384/512 ... yes 320/514 ... yes 388/515 ... yes 275/516 ... yes 24/517 ... yes 81/519 ... yes 47/520 ... yes 376/521 ... yes 47/522 ... yes 101/527 ... yes 143/529 ... yes 42/530 ... yes 143/531 ... yes 405/532 ... yes 409/533 ... yes 411/534 ... yes 410/535 ... yes 403/536 ... yes 413/537 ... yes 412/538 ... yes 414/539 ... yes 415/540 ... yes 308/541 ... yes 143/542 ... yes 275/543 ... yes 421/544 ... yes 473/545 ... yes 101/546 ... yes 476/547 ... yes 9/549 ... yes 47/550 ... yes 47/551 ... yes 425/552 ... yes 47/553 ... yes 226/554 ... yes 425/555 ... yes 425/556 ... yes 239/557 ... yes 425/558 ... yes 426/559 ... yes 426/560 ... yes 425/561 ... yes 425/562 ... yes 425/563 ... yes 471/564 ... yes 426/565 ... yes 427/566 ... yes 425/567 ... yes 47/568 ... yes 81/569 ... yes 431/570 ... yes 430/571 ... yes 375/572 ... yes 375/573 ... yes 435/574 ... yes 47/575 ... yes 435/576 ... yes 47/577 ... yes 47/578 ... yes 47/579 ... yes 47/580 ... yes 47/581 ... yes 47/582 ... yes 47/583 ... yes 47/584 ... yes 307/585 ... yes 275/587 ... yes 435/588 ... yes 435/589 ... yes 106/591 ... yes 9/593 ... yes 471/596 ... yes 2/597 ... yes 115/598 ... yes 443/599 ... yes 275/600 ... yes 443/601 ... yes 443/602 ... yes 275/603 ... yes 185/604 ... yes 338/605 ... yes 275/606 ... yes 275/607 ... yes 375/608 ... yes 375/609 ... yes 205/610 ... yes 61/611 ... yes 475/612 ... yes 239/613 ... yes 459/614 ... yes 275/615 ... yes 430/616 ... yes 101/617 ... yes 275/618 ... yes 101/619 ... yes 435/620 ... yes 469/623 ... yes 469/624 ... yes 469/625 ... yes 469/626 ... yes 469/627 ... yes 469/628 ... yes 469/629 ... yes 469/630 ... yes 469/631 ... yes 469/632 ... yes 469/633 ... yes 469/634 ... yes 469/635 ... yes 469/636 ... yes 469/638 ... yes 469/639 ... yes 469/640 ... yes 469/641 ... yes 468/643 ... yes 469/644 ... yes 468/645 ... yes 75/646 ... yes 472/647 ... yes 471/648 ... yes 275/650 ... yes 470/652 ... yes 370/653 ... yes 275/655 ... yes 75/656 ... yes 471/657 ... yes 475/658 ... yes 470/659 ... yes 475/660 ... yes 473/661 ... yes 473/662 ... yes 470/663 ... yes 473/664 ... yes 470/666 ... yes 471/667 ... yes 75/668 ... yes 473/670 ... yes 473/671 ... yes 473/672 ... yes 470/673 ... yes 470/674 ... yes 471/675 ... yes 481/676 ... yes 482/677 ... yes 482/678 ... yes 482/679 ... yes 482/680 ... yes 476/681 ... yes 471/682 ... yes 492/684 ... yes 470/685 ... yes 275/687 ... yes 497/688 ... yes 492/689 ... yes 492/690 ... yes 155/691 ... yes 9/692 ... yes 500/693 ... yes 500/694 ... yes 101/695 ... yes 47/696 ... yes 471/697 ... yes 500/698 ... yes 503/699 ... yes 503/700 ... yes 503/701 ... yes 503/702 ... yes 503/703 ... yes 492/704 ... yes 503/705 ... yes 503/706 ... yes 512/707 ... yes 231/708 ... yes 500/711 ... yes 83/712 ... yes 513/715 ... yes 513/716 ... yes 515/717 ... yes 24/718 ... yes 239/719 ... yes 396/721 ... yes 376/722 ... yes 375/723 ... yes 83/724 ... yes 519/725 ... yes 503/726 ... yes 503/727 ... yes 9/728 ... yes 9/729 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.3) Git version >= 2.21.0 ? ... yes (2.21.0) Git user has default SSH configuration? ... yes Active users: ... 430 Elasticsearch version 5.6 - 6.x? ... skipped (elasticsearch is disabled) Checking GitLab App ... Finished Checking GitLab subtasks ... Finished
Possible fixes
(If you can, link to the line of code that might be responsible for the problem)
Edited by siim