autoscaling gitlab runner on aws ec2 logs "error during connect"
Summary
I've created an autoscaling gitlab runner loosely based on: https://about.gitlab.com/blog/2017/11/23/autoscale-ci-runners/
When a pipeline is run in GitLab, an EC2 runner builder instance is created but the pipeline logs:
Running with gitlab-runner 14.7.0 (98daeee0)
on aws-build-linux-docker-large-autoscaler fWgdgZy-
Preparing the "docker+machine" executor
ERROR: Failed to remove network for build
ERROR: Preparation failed: error during connect: Get "https://10.143.64.89:2376/v1.25/info": Forbidden (docker.go:853:0s)
I've enabled debug in the runner manager instance which logs:
Created docker-machine ^[[0;m created^[[0;m=2022-01-20 12:39:18.357503549 +0000 UTC m=+159.474788823 docker^[[0;m=tcp://10.143.64.88:2376 job^[[0;m=323706 name^[[0;m=runner-fwgdgzy--runner-1642682358-43731d30 now^[[0;m=2022-01-20 12:41:07.985741895 +0000 UTC m=+269.103027191 project^[[0;m=893 runner^[[0;m=fWgdgZy- usedcount^[[0;m=1
^[[37;1mFIXME: Got an status-code for which error does not match any expected type!!!: -1^[[0;m ^[[37;1mmodule^[[0;m=api ^[[37;1mstatus_code^[[0;m=-1
Preparing docker-machine wrapped executor failed ^[[0;m created^[[0;m=2022-01-20 12:39:18.357503549 +0000 UTC m=+159.474788823 docker^[[0;m=tcp://10.143.64.88:2376 job^[[0;m=323706 name^[[0;m=runner-fwgdgzy--runner-1642682358-43731d30 now^[[0;m=2022-01-20 12:41:08.02958722 +0000 UTC m=+269.146872614 project^[[0;m=893 runner^[[0;m=fWgdgZy- usedcount^[[0;m=1
^[[37;1mVolumes manager is empty, skipping volumes cleanup^[[0;m ^[[37;1mjob^[[0;m=323706 ^[[37;1mproject^[[0;m=893 ^[[37;1mrunner^[[0;m=fWgdgZy-
^[[31;1mERROR: Failed Connection to 10.143.64.80 closed by remote host.;1merror^[[0;m=networksManager is undefined ^[[31;1mjob^[[0;m=323706 ^[[31;1mnetwork^[[0;m= ^[[31;1mp
Connection to 10.143.64.80 closed.0;m=fWgdgZy-
Cleaned up docker-machine ^[[0;m created^[[0;m=2022-01-20 12:39:18.357503549 +0000 UTC m=+159.474788823 docker^[[0;m=tcp://10.143.64.88:2376 job^[[
^[[0;m=runner-fwgdgzy--runner-1642682358-43731d30 now^[[0;m=2022-01-20 12:41:08.031246566 +0000 UTC m=+269.148531929 project^[[0;m=893 runner^[[0;m=fWgdgZy-
WARNING: Preparation failed: error during connect: Get "https://10.143.64.88:2376/v1.25/info": Forbidden (docker.go:853:0s)^[[0;m ^[[0;33mjob^[[0;m=323706 ^[[0;33m
config.toml:
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "aws-build-linux-docker-large-autoscaler"
limit = 6
url = "https://..."
token = "$GITLAB_TOKEN"
executor = "docker+machine"
[runners.custom_build_dir]
enabled = true
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
image = "ubuntu:focal"
privileged = true
disable_cache = false
volumes = ["/cache", "/var/run/docker.sock:/var/run/docker.sock"]
shm_size = 0
[runners.machine]
IdleCount = 0
MachineDriver = "amazonec2"
MachineName = "runner-%s"
MachineOptions = ["amazonec2-access-key=${var.aws_access_key_id}", "amazonec2-secret-key=${var.aws_secret_access_key}", "amazonec2-ssh-user=ubuntu", "amazonec2-region=${var.region}", "amazonec2-instance-type=t2.micro", "amazonec2-vpc-id=vpc-...", "amazonec2-subnet-id=subnet-...", "amazonec2-use-private-address=true", "amazonec2-zone=c", "amazonec2-request-spot-instance=true", "amazonec2-spot-price="]
IdleTime = 30