digitalocean: Error creating machine: Error in driver during machine creation: too many tries (60)
We have an autoscaled gitlab runner setup in DO. It was working fine for many months until the last week. We didn't change anything on our side, but now we have "Error in driver during machine creation: too many tries (60)" every time we try to create a new machine. Manually or with gitlab-runner, doesn't matter. I can create a new droplet with plain API request. I have tried to reach DO tech support, but all they can told me is "something wrong with docker-machine, use terraform".
My config.toml:
concurrent = 10
check_interval = 10
[session_server]
session_timeout = 1800
[[runners]]
name = "do-autoscale"
limit = 5
output_limit = 10000
url = "https://gitlab.com/"
token = "XXX"
executor = "docker+machine"
[runners.custom_build_dir]
[runners.cache]
Type = "s3"
Path = "cache"
Shared = true
[runners.cache.s3]
ServerAddress = "XXX.sfo2.digitaloceanspaces.com"
AccessKey = "XXX"
SecretKey = "XXX"
BucketName = "XXX"
[runners.docker]
host = "tcp://docker:2375"
tls_verify = true
image = "alpine"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache", "/certs/client"]
cache_dir = "/cache"
pull_policy = ["always"]
shm_size = 0
[runners.machine]
IdleCount = 0
IdleTime = 600
MachineDriver = "digitalocean"
MachineName = "gitlab-runner-autoscale-%s"
MachineOptions = ["digitalocean-image=ubuntu-18-04-x64", "digitalocean-ssh-user=root", "digitalocean-access-token=XXX", "digitalocean-region=sfo2", "digitalocean-size=s-2vcpu-4gb", "digitalocean-tags=gitlab-runner-autoscale", "digitalocean-private-networking=true", "digitalocean-monitoring=false", "engine-install-url=https://releases.rancher.com/install-docker/19.03.9.sh"]
Manual:
root@nft-grb:/etc/gitlab-runner# docker-machine create --driver digitalocean --digitalocean-image ubuntu-20-04-x64 --digitalocean-access-token $DOTOKEN --digitalocean-region fra1 --engine-install-url "https://releases.rancher.com/install-docker/19.03.9.sh" docker-ubuntu-2004
Running pre-create checks...
Creating machine...
(docker-ubuntu-2004) Creating SSH key...
(docker-ubuntu-2004) Creating Digital Ocean droplet...
(docker-ubuntu-2004) Waiting for IP address to be assigned to the Droplet...
(docker-ubuntu-2004) Droplet creation failed on Digital Ocean, removing...
Error creating machine: Error in driver during machine creation: too many tries (60)
I use latest docker-machine:
root@nft-grb:~# docker-machine version
docker-machine version 0.16.2-gitlab.13, build 8b93db11
Any ideas what might be wrong?