Skip to content

Docker-machine executor tries to provision

Summary

Docker-machine executor try to create all required machine in Autoscaler and IdleCount this behavior cause impact on infrastructure and rate-limit on public cloud providers, for example if we use IdleCount=50 the runner start to creating all these 50 instances concurrently and this cause 429 rate-limit on public clouds like digitalocean, Hetzner and etc.

Also on hosted infrastructures like vmware these concurrent creation put huge impact on clusters and cause functionality problems.

Steps to reproduce

  1. Use docker+machine executor and digitalocean driver
  2. set the IdelClount=50
  3. Wait
  4. You will see lots off ratelimits/429 logs

Runner will create macines then can't connect them because of rate-limit and left these machines as an Error, this loop continues until the account reach the limit of droplets.

Tested on `hetzner` too.

Actual behavior

Expected behavior

The machine creation should have wait between creation or do the creation nonsimultaneous.

Relevant logs and/or screenshots

runner log 
ERROR: Error with pre-create check: "GET https://api.digitalocean.com/v2/regions: 429 Too many requests"  driver=digitalocean name=runner-p7bqgpld-do-fra1-runner1062-1703055592-f24dd6fe operation=create
ERROR: Machine creation failed                      error=exit status 3 name=runner-p7bqgpld-do-fra1-runner1062-1703055592-f24dd6fe time=2.49333776s
WARNING: Requesting machine removal                 lifetime=2.493655253s name=runner-p7bqgpld-do-fra1-runner1062-1703055592-f24dd6fe now=2023-12-20 06:59:54.7662291 +0000 UTC m=+30032.808018372 reason=Failed to create used=2.493655839s usedCount=0
WARNING: Skipping machine removal, because it doesn't exist  lifetime=2.493755152s name=runner-p7bqgpld-do-fra1-runner1062-1703055592-f24dd6fe reason=Failed to create used=48.922µs usedCount=0
Machine removed                                     lifetime=2.493825726s name=runner-p7bqgpld-do-fra1-runner1062-1703055592-f24dd6fe now=2023-12-20 06:59:54.766395974 +0000 UTC m=+30032.808185226 reason=Failed to create retries=0 used=119.13µs usedCount=0
ERROR: Error with pre-create check: "GET https://api.digitalocean.com/v2/regions: 429 Too many requests"  driver=digitalocean name=runner-p7bqgpld-do-fra1-runner1062-1703055592-aefe64d0 operation=create
WARNING: Problem while reading command output       error=read |0: file already closed
WARNING: Problem while reading command output       error=read |0: file already closed
ERROR: Machine creation failed                      error=exit status 3 name=runner-p7bqgpld-do-fra1-runner1062-1703055592-aefe64d0 time=2.491628989s
WARNING: Requesting machine removal                 lifetime=2.498636954s name=runner-p7bqgpld-do-fra1-runner1062-1703055592-aefe64d0 now=2023-12-20 06:59:54.771113981 +0000 UTC m=+30032.812903233 reason=Failed to create used=2.498637287s usedCount=0
WARNING: Skipping machine removal, because it doesn't exist  lifetime=2.498827736s name=runner-p7bqgpld-do-fra1-runner1062-1703055592-aefe64d0 reason=Failed to create used=166.276µs usedCount=0
Machine removed                                     lifetime=2.499257159s name=runner-p7bqgpld-do-fra1-runner1062-1703055592-aefe64d0 now=2023-12-20 06:59:54.771734237 +0000 UTC m=+30032.813523484 reason=Failed to create retries=0 used=595.558µs usedCount=0
ERROR: Error with pre-create check: "GET https://api.digitalocean.com/v2/regions: 429 Too many requests"  driver=digitalocean name=runner-p7bqgpld-do-fra1-runner1062-1703055592-076ad97b operation=create
ERROR: Machine creation failed                      error=exit status 3 name=runner-p7bqgpld-do-fra1-runner1062-1703055592-076ad97b time=2.511556933s
WARNING: Requesting machine removal                 lifetime=2.512279384s name=runner-p7bqgpld-do-fra1-runner1062-1703055592-076ad97b now=2023-12-20 06:59:54.784717753 +0000 UTC m=+30032.826506999 reason=Failed to create used=2.512279875s usedCount=0
ERROR: Error with pre-create check: "GET https://api.digitalocean.com/v2/regions: 429 Too many requests"  driver=digitalocean name=runner-p7bqgpld-do-fra1-runner1062-1703055592-6c06af91 operation=create
doctl output 
doctl compute ssh-key list | grep runner-
40414052    runner-p7bqgpld-do-fra1-runner1062-1703055592-993ba2ea         0f:dc:4e:14:55:14:56:66:c1:68:6f:59:1e:9c:48:c9
40414051    runner-p7bqgpld-do-fra1-runner1062-1703055592-1e968652         f9:47:7b:35:63:9c:f1:74:3e:92:e8:4c:38:6a:37:3e
40414050    runner-p7bqgpld-do-fra1-runner1062-1703055592-843c50eb         f1:d8:3f:65:61:54:70:49:06:d7:94:7a:b0:de:4b:cb
40414049    runner-p7bqgpld-do-fra1-runner1062-1703055592-f5a40316         4e:f3:bf:80:b9:8b:dd:47:59:a5:de:5f:1f:51:a7:db
40414048    runner-p7bqgpld-do-fra1-runner1062-1703055592-98ed0d50         fb:a5:16:4a:f1:82:53:43:c4:04:88:a6:31:82:1c:4f
40414046    runner-p7bqgpld-do-fra1-runner1062-1703055592-75a4932a         f5:24:5e:f0:2d:27:0e:6c:4b:1f:b5:55:2e:ec:94:42
40414045    runner-p7bqgpld-do-fra1-runner1062-1703055592-e5ec323e         b3:2b:da:7d:6e:24:9c:06:f6:8f:27:1e:c1:59:5e:2a
40414044    runner-p7bqgpld-do-fra1-runner1062-1703055592-e450ff71         1e:ba:e5:4e:f5:ea:92:aa:c7:97:d8:15:fe:b0:40:c9
40414043    runner-p7bqgpld-do-fra1-runner1062-1703055592-68de2dd1         61:9a:3c:41:c3:a0:59:ee:bd:cd:87:52:2b:0d:0c:c1
40414042    runner-p7bqgpld-do-fra1-runner1062-1703055592-ecad61ac         4f:f0:a2:a5:72:dc:e9:fd:0f:f1:32:0b:75:6d:d7:89
40414031    runner-p7bqgpld-do-fra1-runner1062-1703055592-5861563f         63:3b:04:a7:4a:ab:ed:b4:52:f6:01:e5:05:2d:3e:3c
40414030    runner-p7bqgpld-do-fra1-runner1062-1703055592-6b6c3e3a         6c:73:2b:36:3d:fa:a0:ff:7f:ab:9e:1a:62:ee:4d:6a
40414029    runner-p7bqgpld-do-fra1-runner1062-1703055592-58eeb5bc         3f:a3:73:6f:58:90:ca:b0:fa:4d:0c:b8:3c:c9:06:e0
40414028    runner-p7bqgpld-do-fra1-runner1062-1703055592-21ab69be         60:bd:b2:5f:74:5a:e8:a7:b3:46:28:85:45:0d:ed:ce
40414027    runner-p7bqgpld-do-fra1-runner1062-1703055592-eac88485         8d:a9:84:82:a8:2a:98:de:5c:24:b6:74:18:ea:1d:d3
40414026    runner-p7bqgpld-do-fra1-runner1062-1703055592-828ac7ec         20:dd:2d:52:98:b9:e4:8d:dd:be:f8:82:61:fa:5a:60
40414025    runner-p7bqgpld-do-fra1-runner1062-1703055592-7407fa3a         8a:5b:e1:e8:18:be:77:d8:83:cb:62:ec:38:90:ef:53
40414022    runner-p7bqgpld-do-fra1-runner1062-1703055592-774eafb8         ad:16:d4:11:d4:c7:70:88:d8:85:2a:10:b4:73:83:1b
40414021    runner-p7bqgpld-do-fra1-runner1062-1703055592-21e3911b         72:95:52:02:db:a3:11:f6:f1:c3:23:70:19:92:de:3e
40414020    runner-p7bqgpld-do-fra1-runner1062-1703055592-f7bc9cab         1e:20:f9:c9:e1:cc:b3:a9:a6:2d:c6:09:31:25:ec:0c
40414006    runner-p7bqgpld-do-fra1-runner1062-1703055592-4d39f6fb         02:2a:9a:2a:2e:ef:f5:ad:d8:7f:94:9f:37:f8:b8:b0
40414005    runner-p7bqgpld-do-fra1-runner1062-1703055592-87b6cb9a         1f:55:0c:00:fb:63:b0:0b:fc:d3:22:c4:55:58:50:74
40414004    runner-p7bqgpld-do-fra1-runner1062-1703055592-28f3b278         5a:18:6d:b8:27:41:5b:76:fb:34:4f:b8:9b:6c:08:fa
40414003    runner-p7bqgpld-do-fra1-runner1062-1703055592-5eee1766         5c:31:02:8a:07:93:98:6e:0f:89:f6:cd:41:03:da:16
40414002    runner-p7bqgpld-do-fra1-runner1062-1703055592-1943e504         3f:3b:2a:bd:b9:b0:e3:b7:f9:71:ea:c6:50:b7:32:98
40414001    runner-p7bqgpld-do-fra1-runner1062-1703055592-19ab8878         67:ad:7a:6c:ee:c5:03:7f:1e:5a:7f:04:25:b0:5e:82
40414000    runner-p7bqgpld-do-fra1-runner1062-1703055592-50f7370d         9a:a3:8a:fc:cb:f2:6b:5c:e1:db:d5:02:f1:eb:3d:c1
40413999    runner-p7bqgpld-do-fra1-runner1062-1703055592-da2ce3c0         43:59:89:c1:df:5c:f4:71:59:ce:af:b3:60:73:5c:df
40413998    runner-p7bqgpld-do-fra1-runner1062-1703055592-2bccd994         db:b6:a2:29:76:3b:0a:23:c6:f5:38:34:40:14:3e:70
40413968    runner-p7bqgpld-do-fra1-runner1062-1703055592-c6499719         1d:82:b9:e0:c4:c4:f0:49:6a:20:21:04:3c:fc:14:79
40413967    runner-p7bqgpld-do-fra1-runner1062-1703055592-58d98a65         26:7b:f2:09:8d:29:e9:ea:3b:f7:08:7a:40:2b:6e:41
40413966    runner-p7bqgpld-do-fra1-runner1062-1703055592-e25d0bd1         91:62:34:bc:ad:89:e1:c1:91:fb:05:c5:f3:34:56:23
40413965    runner-p7bqgpld-do-fra1-runner1062-1703055592-94706ae0         0c:f7:d0:96:24:50:29:32:ce:59:ae:f4:c4:0f:28:30
40413964    runner-p7bqgpld-do-fra1-runner1062-1703055592-c4882b0b         f9:db:cc:73:1a:eb:6e:5e:07:de:98:a1:59:e0:a5:76
40413963    runner-p7bqgpld-do-fra1-runner1062-1703055592-a853a11c         b5:8d:91:47:e3:f2:77:be:3c:04:90:d4:38:87:62:5f
40413962    runner-p7bqgpld-do-fra1-runner1062-1703055592-6e0a8083         ee:26:f4:bc:cd:df:85:93:1a:38:22:03:ac:b2:00:33
40413961    runner-p7bqgpld-do-fra1-runner1062-1703055592-f24dd6fe         a4:98:53:b5:00:f5:80:42:8f:53:db:06:00:1a:85:91
40413960    runner-p7bqgpld-do-fra1-runner1062-1703055592-aefe64d0         a1:81:69:9f:0b:34:6e:5e:a6:99:4a:1b:3c:6b:54:2d
40413959    runner-p7bqgpld-do-fra1-runner1062-1703055592-076ad97b         eb:89:9b:61:bb:10:21:d1:bc:1b:4e:de:15:5d:8f:49

Environment description

config.toml contents 
concurrent = 50
check_interval = 0

[[runners]]
  name = "test_Parsa"
  url = "https://gitlab.test.com"
  id = 280
  token = "sometoken"
  token_obtained_at = 2023-12-20T07:31:10Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "docker+machine"
  environment = ["DOCKER_TLS_CERTDIR=/certs"]
  [runners.custom_build_dir]
  [runners.docker]
    tls_verify = false
    image = "docker:stable"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = true
    disable_cache = false
    volumes = ["/certs/client", "/var/run/docker.sock:/var/run/docker.sock", "/cache"]
    pull_policy = ["always", "if-not-present"]
    shm_size = 0
  [runners.machine]
    MaxGrowthRate = 50
    IdleCount = 50
    IdleScaleFactor = 0.0
    IdleCountMin = 0
    IdleTime = 300
    MaxBuilds = 10
    MachineDriver = "digitalocean"
    MachineName = "TEST-PARSA-%s"
    MachineOptions = ["digitalocean-image=docker-20-04", "digitalocean-ssh-user=root", "digitalocean-access-token=do_access_token", "digitalocean-region=ams3", "digitalocean-size=s-4vcpu-8gb", "digitalocean-userdata=/etc/gitlab-runner/userdata.txt"]

Used GitLab Runner version

arch=amd64 os=linux pid=x revision=7178588dversion=15.5.1

Possible fixes

!4544 (closed)

Edited by Parsa Yousefi