Docker+machine executor creating too many instances

Summary

Using then docker+machine executor creates too many machines:

  • When the limit is set to 4, it initially creates 5 machine instances and then scales down to 4.
  • When the limit is set to 1, it initially creates 2 machine instances and then scales down to 1.
  • When the limit is not set, it creates 2 machine instances.

Steps to reproduce (limit = 4)

Save the following in gitlab-runner/config.toml:

concurrent = 10
check_interval = 1

[[runners]]
  name = "runner-digitalocean-0"
  limit = 4
  url = "https://<gitlab-host>"
  token = "cc70ebba48ed1b288af2..."
  executor = "docker+machine"
  [runners.docker]
    tls_verify = false
    image = "mediamoose/inner-ci:1.7"
    privileged = false
    disable_cache = true
    volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
  [runners.cache]
  [runners.machine]
    IdleCount = 1
    IdleTime = 3500
    MaxBuilds = 20
    MachineDriver = "digitalocean"
    MachineName = "autoscale-%s"
    MachineOptions = [
      "digitalocean-image=coreos-stable",
      "digitalocean-ssh-user=core",
      "digitalocean-access-token=<do-token>",
      "digitalocean-region=ams2",
      "digitalocean-size=512mb",
      "digitalocean-private-networking"
    ]
    OffPeakTimezone = ""
    OffPeakIdleCount = 0
    OffPeakIdleTime = 0

Apply docker-machine certificate generation workaround by mentioned in https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/2184 -> https://github.com/docker/machine/issues/4005:

docker run -it --rm -v $PWD/docker:/root/.docker --entrypoint docker-machine gitlab/gitlab-runner:alpine-v1.11.1 create -d none --url localhost test
docker run -it --rm -v $PWD/docker:/root/.docker --entrypoint docker-machine gitlab/gitlab-runner:alpine-v1.11.1 rm -f test

Start the runner:

docker run -it --rm \
    -v $PWD/gitlab-runner:/etc/gitlab-runner \
    -v $PWD/docker:/root/.docker \
    gitlab/gitlab-runner:alpine-v1.11.1 

Actual behavior

5 Instances are getting created.

doctl compute droplet list

ID          Name                                            Public IPv4       Public IPv6   Memory  VCPUs  Disk   Region  Image                       Status  Tags
45168796    runner-cc70ebba-autoscale-1491658620-bf81dd4f   146.185.172.147                 512     1      20     ams2    CoreOS 1298.7.0 (stable)    new
45168803    runner-cc70ebba-autoscale-1491658622-fa6a6fc1   188.226.144.132                 512     1      20     ams2    CoreOS 1298.7.0 (stable)    new
45168804    runner-cc70ebba-autoscale-1491658619-120b14f6   188.226.144.133                 512     1      20     ams2    CoreOS 1298.7.0 (stable)    new
45168806    runner-cc70ebba-autoscale-1491658621-55279a9f   188.226.144.134                 512     1      20     ams2    CoreOS 1298.7.0 (stable)    new
45168807    runner-cc70ebba-autoscale-1491658618-cd162208   188.226.144.135                 512     1      20     ams2    CoreOS 1298.7.0 (stable)    new

After a while one instance gets removed

ID          Name                                            Public IPv4       Public IPv6   Memory  VCPUs  Disk   Region  Image                       Status  Tags
45168796    runner-cc70ebba-autoscale-1491658620-bf81dd4f   146.185.172.147                 512     1      20     ams2    CoreOS 1298.7.0 (stable)    new
45168804    runner-cc70ebba-autoscale-1491658619-120b14f6   188.226.144.133                 512     1      20     ams2    CoreOS 1298.7.0 (stable)    new
45168806    runner-cc70ebba-autoscale-1491658621-55279a9f   188.226.144.134                 512     1      20     ams2    CoreOS 1298.7.0 (stable)    new
45168807    runner-cc70ebba-autoscale-1491658618-cd162208   188.226.144.135                 512     1      20     ams2    CoreOS 1298.7.0 (stable)    new

Expected behavior

Just one DigitalOcean instance getting created (conform IdleCount = 1).

Relevant logs and/or screenshots

GitLab runner logs:

Creating CA: /root/.docker/machine/certs/ca.pem
Creating client certificate: /root/.docker/machine/certs/cert.pem
Running pre-create checks...
Creating machine...
To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: docker-machine env test
About to remove test
WARNING: This action will delete both local reference and remote instance.
Successfully removed test
Running in system-mode.

Registering runner... succeeded                     runner=Hf_yxXCU
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
Starting multi-runner from /etc/gitlab-runner/config.toml ...  builds=0
Running in system-mode.

Configuration loaded                                builds=0
Metrics server disabled
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
Running pre-create checks...                        driver=digitalocean name=runner-cc70ebba-autoscale-1491658618-cd162208 operation=create
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
Running pre-create checks...                        driver=digitalocean name=runner-cc70ebba-autoscale-1491658619-120b14f6 operation=create
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
Running pre-create checks...                        driver=digitalocean name=runner-cc70ebba-autoscale-1491658620-bf81dd4f operation=create
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
Running pre-create checks...                        driver=digitalocean name=runner-cc70ebba-autoscale-1491658621-55279a9f operation=create
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
Creating machine...                                 driver=digitalocean name=runner-cc70ebba-autoscale-1491658619-120b14f6 operation=create
Creating machine...                                 driver=digitalocean name=runner-cc70ebba-autoscale-1491658620-bf81dd4f operation=create
(runner-cc70ebba-autoscale-1491658619-120b14f6) Creating SSH key...  driver=digitalocean name=runner-cc70ebba-autoscale-1491658619-120b14f6 operation=create
(runner-cc70ebba-autoscale-1491658620-bf81dd4f) Creating SSH key...  driver=digitalocean name=runner-cc70ebba-autoscale-1491658620-bf81dd4f operation=create
Creating machine...                                 driver=digitalocean name=runner-cc70ebba-autoscale-1491658618-cd162208 operation=create
(runner-cc70ebba-autoscale-1491658618-cd162208) Creating SSH key...  driver=digitalocean name=runner-cc70ebba-autoscale-1491658618-cd162208 operation=create
Creating machine...                                 driver=digitalocean name=runner-cc70ebba-autoscale-1491658621-55279a9f operation=create
(runner-cc70ebba-autoscale-1491658621-55279a9f) Creating SSH key...  driver=digitalocean name=runner-cc70ebba-autoscale-1491658621-55279a9f operation=create
Running pre-create checks...                        driver=digitalocean name=runner-cc70ebba-autoscale-1491658622-fa6a6fc1 operation=create
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
Creating machine...                                 driver=digitalocean name=runner-cc70ebba-autoscale-1491658622-fa6a6fc1 operation=create
(runner-cc70ebba-autoscale-1491658622-fa6a6fc1) Creating SSH key...  driver=digitalocean name=runner-cc70ebba-autoscale-1491658622-fa6a6fc1 operation=create
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
(runner-cc70ebba-autoscale-1491658620-bf81dd4f) Creating Digital Ocean droplet...  driver=digitalocean name=runner-cc70ebba-autoscale-1491658620-bf81dd4f operation=create
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
(runner-cc70ebba-autoscale-1491658620-bf81dd4f) Waiting for IP address to be assigned to the Droplet...  driver=digitalocean name=runner-cc70ebba-autoscale-1491658620-bf81dd4f operation=create
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
Waiting for machine to be running, this may take a few minutes...  driver=digitalocean name=runner-cc70ebba-autoscale-1491658620-bf81dd4f operation=create
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
(runner-cc70ebba-autoscale-1491658622-fa6a6fc1) Creating Digital Ocean droplet...  driver=digitalocean name=runner-cc70ebba-autoscale-1491658622-fa6a6fc1 operation=create
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
(runner-cc70ebba-autoscale-1491658619-120b14f6) Creating Digital Ocean droplet...  driver=digitalocean name=runner-cc70ebba-autoscale-1491658619-120b14f6 operation=create
(runner-cc70ebba-autoscale-1491658622-fa6a6fc1) Waiting for IP address to be assigned to the Droplet...  driver=digitalocean name=runner-cc70ebba-autoscale-1491658622-fa6a6fc1 operation=create
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
(runner-cc70ebba-autoscale-1491658619-120b14f6) Waiting for IP address to be assigned to the Droplet...  driver=digitalocean name=runner-cc70ebba-autoscale-1491658619-120b14f6 operation=create
(runner-cc70ebba-autoscale-1491658621-55279a9f) Creating Digital Ocean droplet...  driver=digitalocean name=runner-cc70ebba-autoscale-1491658621-55279a9f operation=create
(runner-cc70ebba-autoscale-1491658618-cd162208) Creating Digital Ocean droplet...  driver=digitalocean name=runner-cc70ebba-autoscale-1491658618-cd162208 operation=create
Waiting for machine to be running, this may take a few minutes...  driver=digitalocean name=runner-cc70ebba-autoscale-1491658622-fa6a6fc1 operation=create
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
(runner-cc70ebba-autoscale-1491658621-55279a9f) Waiting for IP address to be assigned to the Droplet...  driver=digitalocean name=runner-cc70ebba-autoscale-1491658621-55279a9f operation=create
(runner-cc70ebba-autoscale-1491658618-cd162208) Waiting for IP address to be assigned to the Droplet...  driver=digitalocean name=runner-cc70ebba-autoscale-1491658618-cd162208 operation=create
Waiting for machine to be running, this may take a few minutes...  driver=digitalocean name=runner-cc70ebba-autoscale-1491658621-55279a9f operation=create
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
Waiting for machine to be running, this may take a few minutes...  driver=digitalocean name=runner-cc70ebba-autoscale-1491658619-120b14f6 operation=create
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
Waiting for machine to be running, this may take a few minutes...  driver=digitalocean name=runner-cc70ebba-autoscale-1491658618-cd162208 operation=create
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
Detecting operating system of created instance...   driver=digitalocean name=runner-cc70ebba-autoscale-1491658620-bf81dd4f operation=create
Waiting for SSH to be available...                  driver=digitalocean name=runner-cc70ebba-autoscale-1491658620-bf81dd4f operation=create
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
Detecting operating system of created instance...   driver=digitalocean name=runner-cc70ebba-autoscale-1491658622-fa6a6fc1 operation=create
Waiting for SSH to be available...                  driver=digitalocean name=runner-cc70ebba-autoscale-1491658622-fa6a6fc1 operation=create
Detecting operating system of created instance...   driver=digitalocean name=runner-cc70ebba-autoscale-1491658618-cd162208 operation=create
Waiting for SSH to be available...                  driver=digitalocean name=runner-cc70ebba-autoscale-1491658618-cd162208 operation=create
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
Detecting operating system of created instance...   driver=digitalocean name=runner-cc70ebba-autoscale-1491658619-120b14f6 operation=create
Waiting for SSH to be available...                  driver=digitalocean name=runner-cc70ebba-autoscale-1491658619-120b14f6 operation=create
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
Detecting operating system of created instance...   driver=digitalocean name=runner-cc70ebba-autoscale-1491658621-55279a9f operation=create
Waiting for SSH to be available...                  driver=digitalocean name=runner-cc70ebba-autoscale-1491658621-55279a9f operation=create
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
Detecting the provisioner...                        driver=digitalocean name=runner-cc70ebba-autoscale-1491658621-55279a9f operation=create
Provisioning with coreOS...                         driver=digitalocean name=runner-cc70ebba-autoscale-1491658621-55279a9f operation=create
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
Copying certs to the local machine directory...     driver=digitalocean name=runner-cc70ebba-autoscale-1491658621-55279a9f operation=create
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
Copying certs to the remote machine...              driver=digitalocean name=runner-cc70ebba-autoscale-1491658621-55279a9f operation=create
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
Setting Docker configuration on the remote daemon...  driver=digitalocean name=runner-cc70ebba-autoscale-1491658621-55279a9f operation=create
Detecting the provisioner...                        driver=digitalocean name=runner-cc70ebba-autoscale-1491658620-bf81dd4f operation=create
Provisioning with coreOS...                         driver=digitalocean name=runner-cc70ebba-autoscale-1491658620-bf81dd4f operation=create
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
Copying certs to the local machine directory...     driver=digitalocean name=runner-cc70ebba-autoscale-1491658620-bf81dd4f operation=create
Checking connection to Docker...                    driver=digitalocean name=runner-cc70ebba-autoscale-1491658621-55279a9f operation=create
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
Docker is up and running!                           driver=digitalocean name=runner-cc70ebba-autoscale-1491658621-55279a9f operation=create
To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: docker-machine env runner-cc70ebba-autoscale-1491658621-55279a9f  driver=digitalocean name=runner-cc70ebba-autoscale-1491658621-55279a9f operation=create
Machine created                                     name=runner-cc70ebba-autoscale-1491658621-55279a9f now=2017-04-08 13:38:11.942516598 +0000 UTC retries=0 time=1m10.198590852s
Detecting the provisioner...                        driver=digitalocean name=runner-cc70ebba-autoscale-1491658618-cd162208 operation=create
Detecting the provisioner...                        driver=digitalocean name=runner-cc70ebba-autoscale-1491658622-fa6a6fc1 operation=create
Provisioning with coreOS...                         driver=digitalocean name=runner-cc70ebba-autoscale-1491658618-cd162208 operation=create
Provisioning with coreOS...                         driver=digitalocean name=runner-cc70ebba-autoscale-1491658622-fa6a6fc1 operation=create
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
Copying certs to the remote machine...              driver=digitalocean name=runner-cc70ebba-autoscale-1491658620-bf81dd4f operation=create
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
Detecting the provisioner...                        driver=digitalocean name=runner-cc70ebba-autoscale-1491658619-120b14f6 operation=create
Copying certs to the local machine directory...     driver=digitalocean name=runner-cc70ebba-autoscale-1491658618-cd162208 operation=create
Copying certs to the local machine directory...     driver=digitalocean name=runner-cc70ebba-autoscale-1491658622-fa6a6fc1 operation=create
Provisioning with coreOS...                         driver=digitalocean name=runner-cc70ebba-autoscale-1491658619-120b14f6 operation=create
Setting Docker configuration on the remote daemon...  driver=digitalocean name=runner-cc70ebba-autoscale-1491658620-bf81dd4f operation=create
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
Copying certs to the local machine directory...     driver=digitalocean name=runner-cc70ebba-autoscale-1491658619-120b14f6 operation=create
Copying certs to the remote machine...              driver=digitalocean name=runner-cc70ebba-autoscale-1491658618-cd162208 operation=create
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
Checking connection to Docker...                    driver=digitalocean name=runner-cc70ebba-autoscale-1491658620-bf81dd4f operation=create
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
WARNING: Failed to update executor docker+machine for cc70ebba No free machines that can process builds
Docker is up and running!                           driver=digitalocean name=runner-cc70ebba-autoscale-1491658620-bf81dd4f operation=create
To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: docker-machine env runner-cc70ebba-autoscale-1491658620-bf81dd4f  driver=digitalocean name=runner-cc70ebba-autoscale-1491658620-bf81dd4f operation=create
Machine created                                     name=runner-cc70ebba-autoscale-1491658620-bf81dd4f now=2017-04-08 13:38:23.743435358 +0000 UTC retries=0 time=1m22.99967378s
Setting Docker configuration on the remote daemon...  driver=digitalocean name=runner-cc70ebba-autoscale-1491658618-cd162208 operation=create
Copying certs to the remote machine...              driver=digitalocean name=runner-cc70ebba-autoscale-1491658622-fa6a6fc1 operation=create
Copying certs to the remote machine...              driver=digitalocean name=runner-cc70ebba-autoscale-1491658619-120b14f6 operation=create
Checking connection to Docker...                    driver=digitalocean name=runner-cc70ebba-autoscale-1491658618-cd162208 operation=create
Docker is up and running!                           driver=digitalocean name=runner-cc70ebba-autoscale-1491658618-cd162208 operation=create
To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: docker-machine env runner-cc70ebba-autoscale-1491658618-cd162208  driver=digitalocean name=runner-cc70ebba-autoscale-1491658618-cd162208 operation=create
Machine created                                     name=runner-cc70ebba-autoscale-1491658618-cd162208 now=2017-04-08 13:38:29.136366444 +0000 UTC retries=0 time=1m30.393479012s
Setting Docker configuration on the remote daemon...  driver=digitalocean name=runner-cc70ebba-autoscale-1491658622-fa6a6fc1 operation=create
Setting Docker configuration on the remote daemon...  driver=digitalocean name=runner-cc70ebba-autoscale-1491658619-120b14f6 operation=create
Checking connection to Docker...                    driver=digitalocean name=runner-cc70ebba-autoscale-1491658619-120b14f6 operation=create
Checking connection to Docker...                    driver=digitalocean name=runner-cc70ebba-autoscale-1491658622-fa6a6fc1 operation=create
Docker is up and running!                           driver=digitalocean name=runner-cc70ebba-autoscale-1491658619-120b14f6 operation=create
To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: docker-machine env runner-cc70ebba-autoscale-1491658619-120b14f6  driver=digitalocean name=runner-cc70ebba-autoscale-1491658619-120b14f6 operation=create
Machine created                                     name=runner-cc70ebba-autoscale-1491658619-120b14f6 now=2017-04-08 13:38:34.036598651 +0000 UTC retries=0 time=1m34.201936225s
Docker is up and running!                           driver=digitalocean name=runner-cc70ebba-autoscale-1491658622-fa6a6fc1 operation=create
To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: docker-machine env runner-cc70ebba-autoscale-1491658622-fa6a6fc1  driver=digitalocean name=runner-cc70ebba-autoscale-1491658622-fa6a6fc1 operation=create
Machine created                                     name=runner-cc70ebba-autoscale-1491658622-fa6a6fc1 now=2017-04-08 13:38:34.363493023 +0000 UTC retries=0 time=1m31.619158895s
WARNING: Removing machine                           created=1m31.809293888s name=runner-cc70ebba-autoscale-1491658622-fa6a6fc1 now=2017-04-08 13:38:34.5534398 +0000 UTC reason=Too many machines used=189.954857ms
About to remove runner-cc70ebba-autoscale-1491658622-fa6a6fc1  name=runner-cc70ebba-autoscale-1491658622-fa6a6fc1 operation=remove
WARNING: This action will delete both local reference and remote instance.  name=runner-cc70ebba-autoscale-1491658622-fa6a6fc1 operation=remove
Successfully removed runner-cc70ebba-autoscale-1491658622-fa6a6fc1  name=runner-cc70ebba-autoscale-1491658622-fa6a6fc1 operation=remove
Machine removed                                     created=1m33.393700206s name=runner-cc70ebba-autoscale-1491658622-fa6a6fc1 now=2017-04-08 13:38:36.137840508 +0000 UTC reason=Too many machines retries=0 used=1.584287s

Environment description

Shared runner on 8.17.5.

Executor: docker+machine

Multirunner running on Docker 1.12.6:

Server:
 Version:      1.12.6
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   d5236f0
 Built:        Tue Feb 28 00:07:14 2017
 OS/Arch:      linux/amd64

Used GitLab Runner version

GitLab runner image: gitlab/gitlab-runner:alpine-v1.11.1 (image-id: 34e9d5d8885f) (docker-machine 0.10.0 is included in this image)

Version:      1.11.1
Git revision: a67a225
Git branch:   1-11-stable
GO version:   go1.7.5
Built:        Fri, 03 Mar 2017 12:35:58 +0000
OS/Arch:      linux/amd64

Temporary workaround

Scaling down by setting limit to 1. Now it scales down to 1 instance. Setting the limit back to 4 does not cause a direct upscale.