Docker+machine suddenly created 20 new machines
Summary
After checking if one docker machine was still available (which wasn't), gitlab-runner attempted to remove it and then created 20 new one (10 instantly, then 10 other within two minutes).
Steps to reproduce
Can't reproduce it, it happens once randomly.
And it's not related to one job in particular.
Actual behavior
Created 20 VM on our ESXi which nearly saturated.
Expected behavior
Should have created only one new machine.
Relevant logs
Here is the relevant part. If requested, I can provide before and after.
I've included the start of two job 10 minutes before the incident, but I don't think they are relevant. 7026 eventually finished after 20 minutes (should take 10). And I had to cancel 7028 when I eventually killed all docker VM manually.
Relevant /var/log/messages part
...
Jun 28 10:43:59 docker02 gitlab-runner[13099]: Checking for jobs... received job=7026 repo_url=http://git.localdomain/product/web.git runner=6x3v2poW
Jun 28 10:44:00 docker02 gitlab-runner[13099]: Checking for jobs... received job=7028 repo_url=http://git.localdomain/product/web.git runner=6x3v2poW
Jun 28 10:44:22 docker02 gitlab-runner[13099]: Starting docker-machine build... created=2019-06-28 08:50:18.137339036 +0200 CEST docker=tcp://10.3.1.206:2376 job=7028 name=runner-6x3v2pow-product-1561704618-9c388276 now=2019-06-28 10:44:22.723146341 +0200 CEST project=42 runner=6x3v2poW usedcount=5
Jun 28 10:44:26 docker02 gitlab-runner[13099]: Starting docker-machine build... created=2019-06-28 09:18:08.224088257 +0200 CEST docker=tcp://10.3.1.208:2376 job=7026 name=runner-6x3v2pow-product-1561706288-fe0fcf3f now=2019-06-28 10:44:26.013957043 +0200 CEST project=42 runner=6x3v2poW usedcount=3
Jun 28 10:54:23 docker02 gitlab-runner[13099]: WARNING: Requesting machine removal lifetime=2h54m15.446557361s name=runner-6x3v2pow-product-1561701608-7dc0c8ca now=2019-06-28 10:54:23.449168765 +0200 CEST reason=machine is unavailable used=1h35m24.890565352s usedCount=2
Jun 28 10:54:23 docker02 gitlab-runner[13099]: WARNING: Stopping machine lifetime=2h54m15.643839877s name=runner-6x3v2pow-product-1561701608-7dc0c8ca reason=machine is unavailable used=192.386626ms usedCount=2
Jun 28 10:54:23 docker02 gitlab-runner[13099]: Running pre-create checks... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-94ee922b operation=create
Jun 28 10:54:24 docker02 gitlab-runner[13099]: Running pre-create checks... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-2b84f88e operation=create
Jun 28 10:54:24 docker02 gitlab-runner[13099]: Running pre-create checks... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-23c169f8 operation=create
Jun 28 10:54:24 docker02 gitlab-runner[13099]: Stopping "runner-6x3v2pow-product-1561701608-7dc0c8ca"... name=runner-6x3v2pow-product-1561701608-7dc0c8ca operation=stop
Jun 28 10:54:24 docker02 gitlab-runner[13099]: Running pre-create checks... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-bc30e61b operation=create
Jun 28 10:54:24 docker02 gitlab-runner[13099]: Running pre-create checks... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-50177b8f operation=create
Jun 28 10:54:24 docker02 gitlab-runner[13099]: Running pre-create checks... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-48b181cb operation=create
Jun 28 10:54:24 docker02 gitlab-runner[13099]: Running pre-create checks... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-dabe7a4b operation=create
Jun 28 10:54:24 docker02 gitlab-runner[13099]: Running pre-create checks... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-b9a85052 operation=create
Jun 28 10:54:33 docker02 gitlab-runner[13099]: Running pre-create checks... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712073-0b08e029 operation=create
Jun 28 10:54:43 docker02 gitlab-runner[13099]: Running pre-create checks... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712083-8527f0ba operation=create
Jun 28 10:54:53 docker02 gitlab-runner[13099]: Running pre-create checks... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712093-a6b20de3 operation=create
Jun 28 10:55:03 docker02 gitlab-runner[13099]: Running pre-create checks... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712103-184261f5 operation=create
Jun 28 10:55:13 docker02 gitlab-runner[13099]: Running pre-create checks... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712113-7d08322b operation=create
Jun 28 10:55:23 docker02 gitlab-runner[13099]: WARNING: Error while stopping machine error=signal: killed lifetime=2h55m15.68995122s name=runner-6x3v2pow-product-1561701608-7dc0c8ca reason=machine is unavailable used=1m0.238493991s usedCount=2
Jun 28 10:55:23 docker02 gitlab-runner[13099]: WARNING: Removing machine lifetime=2h55m15.690758585s name=runner-6x3v2pow-product-1561701608-7dc0c8ca reason=machine is unavailable used=1m0.239301011s usedCount=2
Jun 28 10:55:23 docker02 gitlab-runner[13099]: About to remove runner-6x3v2pow-product-1561701608-7dc0c8ca name=runner-6x3v2pow-product-1561701608-7dc0c8ca operation=remove
Jun 28 10:55:23 docker02 gitlab-runner[13099]: WARNING: This action will delete both local reference and remote instance. name=runner-6x3v2pow-product-1561701608-7dc0c8ca operation=remove
Jun 28 10:55:23 docker02 gitlab-runner[13099]: Running pre-create checks... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712123-ac74bfd5 operation=create
Jun 28 10:55:33 docker02 gitlab-runner[13099]: Running pre-create checks... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712133-cfa6594c operation=create
Jun 28 10:55:43 docker02 gitlab-runner[13099]: Running pre-create checks... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712143-75b40b44 operation=create
Jun 28 10:55:53 docker02 gitlab-runner[13099]: Running pre-create checks... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712153-d42555e9 operation=create
Jun 28 10:56:03 docker02 gitlab-runner[13099]: Running pre-create checks... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712163-fc72f881 operation=create
Jun 28 10:56:13 docker02 gitlab-runner[13099]: Running pre-create checks... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712173-3c0ac14c operation=create
Jun 28 10:56:23 docker02 gitlab-runner[13099]: Running pre-create checks... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712183-a4e9832b operation=create
Jun 28 10:56:28 docker02 gitlab-runner[13099]: Creating machine... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712073-0b08e029 operation=create
Jun 28 10:56:28 docker02 gitlab-runner[13099]: Creating machine... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-b9a85052 operation=create
Jun 28 10:56:29 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712073-0b08e029) Copying /root/.docker/machine/cache/boot2docker.iso to /root/.docker/machine/machines/runner-6x3v2pow-product-1561712073-0b08e029/boot2docker.iso... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712073-0b08e029 operation=create
Jun 28 10:56:30 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712073-0b08e029) Generating SSH Keypair... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712073-0b08e029 operation=create
Jun 28 10:56:30 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712063-b9a85052) Copying /root/.docker/machine/cache/boot2docker.iso to /root/.docker/machine/machines/runner-6x3v2pow-product-1561712063-b9a85052/boot2docker.iso... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-b9a85052 operation=create
Jun 28 10:56:30 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712063-b9a85052) Generating SSH Keypair... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-b9a85052 operation=create
Jun 28 10:56:48 docker02 gitlab-runner[13099]: Creating machine... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-94ee922b operation=create
Jun 28 10:56:48 docker02 gitlab-runner[13099]: Creating machine... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-23c169f8 operation=create
Jun 28 10:56:48 docker02 gitlab-runner[13099]: Creating machine... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-48b181cb operation=create
Jun 28 10:56:48 docker02 gitlab-runner[13099]: Creating machine... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-dabe7a4b operation=create
Jun 28 10:56:48 docker02 gitlab-runner[13099]: Creating machine... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-2b84f88e operation=create
Jun 28 10:56:48 docker02 gitlab-runner[13099]: Creating machine... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-50177b8f operation=create
Jun 28 10:56:48 docker02 gitlab-runner[13099]: Creating machine... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-bc30e61b operation=create
Jun 28 10:56:49 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712063-50177b8f) Copying /root/.docker/machine/cache/boot2docker.iso to /root/.docker/machine/machines/runner-6x3v2pow-product-1561712063-50177b8f/boot2docker.iso... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-50177b8f operation=create
Jun 28 10:56:49 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712063-dabe7a4b) Copying /root/.docker/machine/cache/boot2docker.iso to /root/.docker/machine/machines/runner-6x3v2pow-product-1561712063-dabe7a4b/boot2docker.iso... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-dabe7a4b operation=create
Jun 28 10:56:49 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712063-23c169f8) Copying /root/.docker/machine/cache/boot2docker.iso to /root/.docker/machine/machines/runner-6x3v2pow-product-1561712063-23c169f8/boot2docker.iso... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-23c169f8 operation=create
Jun 28 10:56:49 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712063-94ee922b) Copying /root/.docker/machine/cache/boot2docker.iso to /root/.docker/machine/machines/runner-6x3v2pow-product-1561712063-94ee922b/boot2docker.iso... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-94ee922b operation=create
Jun 28 10:56:49 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712063-48b181cb) Copying /root/.docker/machine/cache/boot2docker.iso to /root/.docker/machine/machines/runner-6x3v2pow-product-1561712063-48b181cb/boot2docker.iso... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-48b181cb operation=create
Jun 28 10:56:49 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712063-2b84f88e) Copying /root/.docker/machine/cache/boot2docker.iso to /root/.docker/machine/machines/runner-6x3v2pow-product-1561712063-2b84f88e/boot2docker.iso... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-2b84f88e operation=create
Jun 28 10:56:49 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712063-bc30e61b) Copying /root/.docker/machine/cache/boot2docker.iso to /root/.docker/machine/machines/runner-6x3v2pow-product-1561712063-bc30e61b/boot2docker.iso... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-bc30e61b operation=create
Jun 28 10:56:50 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712063-50177b8f) Generating SSH Keypair... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-50177b8f operation=create
Jun 28 10:56:50 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712063-23c169f8) Generating SSH Keypair... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-23c169f8 operation=create
Jun 28 10:56:50 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712063-94ee922b) Generating SSH Keypair... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-94ee922b operation=create
Jun 28 10:56:50 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712063-dabe7a4b) Generating SSH Keypair... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-dabe7a4b operation=create
Jun 28 10:56:50 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712063-48b181cb) Generating SSH Keypair... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-48b181cb operation=create
Jun 28 10:56:50 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712063-2b84f88e) Generating SSH Keypair... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-2b84f88e operation=create
Jun 28 10:56:50 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712063-bc30e61b) Generating SSH Keypair... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-bc30e61b operation=create
Jun 28 10:56:52 docker02 gitlab-runner[13099]: Creating machine... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712083-8527f0ba operation=create
Jun 28 10:56:52 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712083-8527f0ba) Copying /root/.docker/machine/cache/boot2docker.iso to /root/.docker/machine/machines/runner-6x3v2pow-product-1561712083-8527f0ba/boot2docker.iso... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712083-8527f0ba operation=create
Jun 28 10:57:33 docker02 gitlab-runner[13099]: Creating machine... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712093-a6b20de3 operation=create
Jun 28 10:57:33 docker02 gitlab-runner[13099]: Creating machine... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712113-7d08322b operation=create
Jun 28 10:57:33 docker02 gitlab-runner[13099]: Creating machine... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712123-ac74bfd5 operation=create
Jun 28 10:57:33 docker02 gitlab-runner[13099]: Creating machine... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712143-75b40b44 operation=create
Jun 28 10:57:33 docker02 gitlab-runner[13099]: Creating machine... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712153-d42555e9 operation=create
Jun 28 10:57:33 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712143-75b40b44) Copying /root/.docker/machine/cache/boot2docker.iso to /root/.docker/machine/machines/runner-6x3v2pow-product-1561712143-75b40b44/boot2docker.iso... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712143-75b40b44 operation=create
Jun 28 10:57:33 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712153-d42555e9) Copying /root/.docker/machine/cache/boot2docker.iso to /root/.docker/machine/machines/runner-6x3v2pow-product-1561712153-d42555e9/boot2docker.iso... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712153-d42555e9 operation=create
Jun 28 10:57:33 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712123-ac74bfd5) Copying /root/.docker/machine/cache/boot2docker.iso to /root/.docker/machine/machines/runner-6x3v2pow-product-1561712123-ac74bfd5/boot2docker.iso... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712123-ac74bfd5 operation=create
Jun 28 10:57:33 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712093-a6b20de3) Copying /root/.docker/machine/cache/boot2docker.iso to /root/.docker/machine/machines/runner-6x3v2pow-product-1561712093-a6b20de3/boot2docker.iso... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712093-a6b20de3 operation=create
Jun 28 10:57:33 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712113-7d08322b) Copying /root/.docker/machine/cache/boot2docker.iso to /root/.docker/machine/machines/runner-6x3v2pow-product-1561712113-7d08322b/boot2docker.iso... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712113-7d08322b operation=create
Jun 28 10:57:56 docker02 gitlab-runner[13099]: Creating machine... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712173-3c0ac14c operation=create
Jun 28 10:57:56 docker02 gitlab-runner[13099]: Creating machine... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712133-cfa6594c operation=create
Jun 28 10:57:56 docker02 gitlab-runner[13099]: Creating machine... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712163-fc72f881 operation=create
Jun 28 10:57:56 docker02 gitlab-runner[13099]: Creating machine... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712103-184261f5 operation=create
Jun 28 10:57:56 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712163-fc72f881) Copying /root/.docker/machine/cache/boot2docker.iso to /root/.docker/machine/machines/runner-6x3v2pow-product-1561712163-fc72f881/boot2docker.iso... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712163-fc72f881 operation=create
Jun 28 10:57:56 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712173-3c0ac14c) Copying /root/.docker/machine/cache/boot2docker.iso to /root/.docker/machine/machines/runner-6x3v2pow-product-1561712173-3c0ac14c/boot2docker.iso... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712173-3c0ac14c operation=create
Jun 28 10:57:56 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712103-184261f5) Copying /root/.docker/machine/cache/boot2docker.iso to /root/.docker/machine/machines/runner-6x3v2pow-product-1561712103-184261f5/boot2docker.iso... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712103-184261f5 operation=create
Jun 28 10:57:56 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712133-cfa6594c) Copying /root/.docker/machine/cache/boot2docker.iso to /root/.docker/machine/machines/runner-6x3v2pow-product-1561712133-cfa6594c/boot2docker.iso... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712133-cfa6594c operation=create
Jun 28 10:57:59 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712063-b9a85052) Creating VM... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-b9a85052 operation=create
Jun 28 10:57:59 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712073-0b08e029) Creating VM... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712073-0b08e029 operation=create
Jun 28 10:57:59 docker02 gitlab-runner[13099]: Creating machine... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712183-a4e9832b operation=create
Jun 28 10:57:59 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712063-2b84f88e) Creating VM... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-2b84f88e operation=create
Jun 28 10:57:59 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712063-23c169f8) Creating VM... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-23c169f8 operation=create
Jun 28 10:57:59 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712063-dabe7a4b) Creating VM... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-dabe7a4b operation=create
Jun 28 10:57:59 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712063-48b181cb) Creating VM... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-48b181cb operation=create
Jun 28 10:57:59 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712063-bc30e61b) Creating VM... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-bc30e61b operation=create
Jun 28 10:57:59 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712063-94ee922b) Creating VM... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-94ee922b operation=create
Jun 28 10:57:59 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712063-50177b8f) Creating VM... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-50177b8f operation=create
Jun 28 10:58:00 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712183-a4e9832b) Copying /root/.docker/machine/cache/boot2docker.iso to /root/.docker/machine/machines/runner-6x3v2pow-product-1561712183-a4e9832b/boot2docker.iso... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712183-a4e9832b operation=create
Jun 28 10:58:18 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712083-8527f0ba) Generating SSH Keypair... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712083-8527f0ba operation=create
Jun 28 10:58:29 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712083-8527f0ba) Creating VM... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712083-8527f0ba operation=create
Jun 28 10:58:51 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712133-cfa6594c) Generating SSH Keypair... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712133-cfa6594c operation=create
Jun 28 10:58:52 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712183-a4e9832b) Generating SSH Keypair... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712183-a4e9832b operation=create
Jun 28 10:59:22 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712133-cfa6594c) Creating VM... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712133-cfa6594c operation=create
Jun 28 10:59:22 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712183-a4e9832b) Creating VM... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712183-a4e9832b operation=create
Jun 28 10:59:25 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712143-75b40b44) Generating SSH Keypair... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712143-75b40b44 operation=create
Jun 28 10:59:25 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712153-d42555e9) Generating SSH Keypair... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712153-d42555e9 operation=create
Jun 28 10:59:25 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712113-7d08322b) Generating SSH Keypair... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712113-7d08322b operation=create
Jun 28 10:59:26 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712123-ac74bfd5) Generating SSH Keypair... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712123-ac74bfd5 operation=create
Jun 28 10:59:26 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712093-a6b20de3) Generating SSH Keypair... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712093-a6b20de3 operation=create
Jun 28 10:59:37 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712163-fc72f881) Generating SSH Keypair... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712163-fc72f881 operation=create
Jun 28 10:59:37 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712103-184261f5) Generating SSH Keypair... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712103-184261f5 operation=create
Jun 28 10:59:37 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712173-3c0ac14c) Generating SSH Keypair... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712173-3c0ac14c operation=create
Jun 28 11:00:12 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712143-75b40b44) Creating VM... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712143-75b40b44 operation=create
Jun 28 11:00:15 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712093-a6b20de3) Creating VM... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712093-a6b20de3 operation=create
Jun 28 11:00:15 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712153-d42555e9) Creating VM... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712153-d42555e9 operation=create
Jun 28 11:00:15 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712123-ac74bfd5) Creating VM... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712123-ac74bfd5 operation=create
Jun 28 11:00:37 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712113-7d08322b) Creating VM... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712113-7d08322b operation=create
Jun 28 11:00:37 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712163-fc72f881) Creating VM... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712163-fc72f881 operation=create
Jun 28 11:00:37 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712173-3c0ac14c) Creating VM... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712173-3c0ac14c operation=create
Jun 28 11:00:37 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712103-184261f5) Creating VM... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712103-184261f5 operation=create
Jun 28 11:01:56 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712063-50177b8f) Uploading Boot2docker ISO ... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-50177b8f operation=create
Jun 28 11:01:56 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712063-2b84f88e) Uploading Boot2docker ISO ... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-2b84f88e operation=create
Jun 28 11:01:56 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712063-23c169f8) Uploading Boot2docker ISO ... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-23c169f8 operation=create
Jun 28 11:01:56 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712063-dabe7a4b) Uploading Boot2docker ISO ... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-dabe7a4b operation=create
Jun 28 11:01:56 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712063-48b181cb) Uploading Boot2docker ISO ... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-48b181cb operation=create
Jun 28 11:01:56 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712063-94ee922b) Uploading Boot2docker ISO ... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-94ee922b operation=create
Jun 28 11:01:56 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712073-0b08e029) Uploading Boot2docker ISO ... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712073-0b08e029 operation=create
Jun 28 11:01:56 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712063-bc30e61b) Uploading Boot2docker ISO ... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-bc30e61b operation=create
Jun 28 11:01:56 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712083-8527f0ba) Uploading Boot2docker ISO ... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712083-8527f0ba operation=create
Jun 28 11:01:56 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712063-b9a85052) Uploading Boot2docker ISO ... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-b9a85052 operation=create
Jun 28 11:03:11 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712063-50177b8f) adding network: VM Network driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-50177b8f operation=create
Jun 28 11:03:11 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712063-50177b8f) Reconfiguring VM driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-50177b8f operation=create
Jun 28 11:03:11 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712063-2b84f88e) adding network: VM Network driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-2b84f88e operation=create
Jun 28 11:03:11 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712063-2b84f88e) Reconfiguring VM driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-2b84f88e operation=create
Jun 28 11:03:12 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712063-94ee922b) adding network: VM Network driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-94ee922b operation=create
Jun 28 11:03:12 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712063-94ee922b) Reconfiguring VM driver=vmwarevsphere name=runner-6x3v2pow-product-1561712063-94ee922b operation=create
Jun 28 11:03:12 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712113-7d08322b) Uploading Boot2docker ISO ... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712113-7d08322b operation=create
Jun 28 11:03:12 docker02 gitlab-runner[13099]: (runner-6x3v2pow-product-1561712103-184261f5) Uploading Boot2docker ISO ... driver=vmwarevsphere name=runner-6x3v2pow-product-1561712103-184261f5 operation=create
Jun 28 11:03:15 docker02 gitlab-runner[13099]: Finished docker-machine build: <nil> created=2019-06-28 09:18:08.224088257 +0200 CEST docker=tcp://10.3.1.208:2376 job=7026 name=runner-6x3v2pow-product-1561706288-fe0fcf3f now=2019-06-28 11:03:15.745569945 +0200 CEST project=42 runner=6x3v2poW usedcount=3
Jun 28 11:03:15 docker02 gitlab-runner[13099]: Job succeeded duration=19m16.357099067s job=7026 project=42 runner=6x3v2poW
...
Environment description
Running gitlab runner our infrastructure, with docker+machine executor, on vmwarevsphere
# docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 1
Server Version: 18.09.6
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: bb71b10fd8f58240ca47fbb579b9d1028eea7c84
runc version: 2b18fe1d885ee5083ef9f0838fee39b62d653e30
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.9.0-9-amd64
Operating System: Debian GNU/Linux 9 (stretch)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.957GiB
Name: docker02
ID: Z5K2:JL42:GMQW:B4BK:SUNZ:JUU4:2GYA:I3PZ:EC74:TWNN:WP6Y:GINK
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
WARNING: No swap limit support
# docker-machine --version
docker-machine version 0.16.0, build 702c267f
config.toml contents
# cat /etc/gitlab-runner/config.toml
concurrent = 10
check_interval = 10
[session_server]
session_timeout = 1800
[[runners]]
name = "docker-autoscale"
limit = 4
url = "http://git.localdomain/"
token = "6x3v2poWmXxeaNW19rRe"
executor = "docker+machine"
[runners.custom_build_dir]
[runners.docker]
tls_verify = false
image = "docker:stable"
privileged = false
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
shm_size = 0
[runners.cache]
Type = "s3"
Shared = true
[runners.cache.s3]
ServerAddress = "10.3.1.113:9000"
AccessKey = "DGR8KO8HJ7QN31SEINO9"
SecretKey = "muK1KMfxeUeFOSdV6OmNPuh4auyJXQm3ZtTGgZFg"
BucketName = "runner"
Insecure = true
[runners.machine]
IdleCount = 1
IdleTime = 14400
MachineDriver = "vmwarevsphere"
MachineName = "product-%s"
MachineOptions = ["vmwarevsphere-username=username", "vmwarevsphere-password=hunter2", "vmwarevsphere-vcenter=esxi02.localdomain", "vmwarevsphere-datastore=ESXDataStore02", "vmwarevsphere-network=VM Network", "vmwarevsphere-memory-size=8192", "engine-insecure-registry=git.localdomain:4567"]
OffPeakPeriods = ["* * 0-7,17-23 * * mon-fri *", "* * * * * sat,sun *"]
OffPeakTimezone = "Europe/Brussels"
OffPeakIdleCount = 0
OffPeakIdleTime = 1200
Used GitLab Runner version
# gitlab-runner --version
Version: 12.0.0
Git revision: 6946bae7
Git branch: 12-0-stable
GO version: go1.8.7
Built: 2019-06-21T15:10:57+0000
OS/Arch: linux/amd64