Gitlab Runner with docker+machine keeps spawning ec2 instances but is unable to use them.
Summary
Gitlab Runner keeps spawning ec2 instances but is unable to use them.
Steps to reproduce
Whenever I call, sudo gitlab-runner run. The docker-machine spawns many instances in AWS until my t2.micro instance number limits are hit. Then it says that there are "No free machines that can process builds".
Before hitting this limit the runner tries to remove instances: and gets the following error:
Stopping "runner-oexvd5xu-gitlab-docker-machine1556215379-6c09b722"... name=runner-oexvd5xu-gitlab-docker-machine1556215379-6c09b722 operation=stop
ERROR: InvalidInstanceID.Malformed: Invalid id: "" name=runner-oexvd5xu-gitlab-docker-machine1556215379-6c09b722 operation=stop
ERROR: status code: 400, request id: 633c9cf7-b6d7-43c1-a768-0457b5303930 name=runner-oexvd5xu-gitlab-docker-machine1556215379-6c09b722 operation=stop
.gitlab-ci.yml
Add the job defection that is failing here
Actual behavior
Expected behavior
Not this.
Relevant logs and/or screenshots
job log
Stopping "runner-oexvd5xu-gitlab-docker-machine1556215379-6c09b722"... name=runner-oexvd5xu-gitlab-docker-machine1556215379-6c09b722 operation=stop
ERROR: InvalidInstanceID.Malformed: Invalid id: "" name=runner-oexvd5xu-gitlab-docker-machine1556215379-6c09b722 operation=stop
ERROR: status code: 400, request id: 633c9cf7-b6d7-43c1-a768-0457b5303930 name=runner-oexvd5xu-gitlab-docker-machine1556215379-6c09b722 operation=stop
WARNING: Error while stopping machine created=190.239914ms error=exit status 1 name=runner-oexvd5xu-gitlab-docker-machine1556215379-6c09b722 reason=Too many builds used=190.172646ms usedCount=1
WARNING: Removing machine created=190.303243ms name=runner-oexvd5xu-gitlab-docker-machine1556215379-6c09b722 reason=Too many builds used=190.235793ms usedCount=1
About to remove runner-oexvd5xu-gitlab-docker-machine1556215379-6c09b722 name=runner-oexvd5xu-gitlab-docker-machine1556215379-6c09b722 operation=remove
WARNING: This action will delete both local reference and remote instance. name=runner-oexvd5xu-gitlab-docker-machine1556215379-6c09b722 operation=remove
Successfully removed runner-oexvd5xu-gitlab-docker-machine1556215379-6c09b722 name=runner-oexvd5xu-gitlab-docker-machine1556215379-6c09b722 operation=remove
(runner-oexvd5xu-gitlab-docker-machine1556215379-6c09b722) Missing instance ID, this is likely due to a failure during machine creation name=runner-oexvd5xu-gitlab-docker-machine1556215379-6c09b722 operation=remove
(runner-oexvd5xu-gitlab-docker-machine1556215379-6c09b722) Missing key pair name, this is likely due to a failure during machine creation name=runner-oexvd5xu-gitlab-docker-machine1556215379-6c09b722 operation=remove
Machine removed created=207.118873ms name=runner-oexvd5xu-gitlab-docker-machine1556215379-6c09b722 now=2019-04-25 18:03:01.461665035 +0000 UTC reason=Too many builds retries=0 used=207.051536ms usedCount=1
WARNING: Requesting machine removal created=8.448µs name=runner-oexvd5xu-gitlab-docker-machine1556215382-965a0248 now=2019-04-25 18:03:04.254986048 +0000 UTC reason=Too many builds used=8.486µs usedCount=1
ERROR: Failed to process runner builds=0 error=failed to update executor: No free machines that can process builds executor=docker+machine runner=oEXVd5xU
WARNING: Stopping machine created=17.247483ms name=runner-oexvd5xu-gitlab-docker-machine1556215382-965a0248 reason=Too many builds used=17.178873ms usedCount=1
Stopping "runner-oexvd5xu-gitlab-docker-machine1556215382-965a0248"... name=runner-oexvd5xu-gitlab-docker-machine1556215382-965a0248 operation=stop
ERROR: InvalidInstanceID.Malformed: Invalid id: "" name=runner-oexvd5xu-gitlab-docker-machine1556215382-965a0248 operation=stop
ERROR: status code: 400, request id: 8ff1dfb1-bd4e-4cff-a6d0-93582d6dbc17 name=runner-oexvd5xu-gitlab-docker-machine1556215382-965a0248 operation=stop
ERROR: Problem while reading command output error=read |0: file already closed
WARNING: Error while stopping machine created=154.420407ms error=exit status 1 name=runner-oexvd5xu-gitlab-docker-machine1556215382-965a0248 reason=Too many builds used=154.351795ms usedCount=1
WARNING: Removing machine created=154.731622ms name=runner-oexvd5xu-gitlab-docker-machine1556215382-965a0248 reason=Too many builds used=154.662849ms usedCount=1
Environment description
config.toml contents
[[runners]]
name = "Group Runner"
url = "https://gitlab.com/"
token = "oEXVxxxxxx"
executor = "docker+machine"
limit = 1
[runners.docker]
image = "nodejs:latest"
privileged = true
disable_cache = true
[runners.machine]
IdleCount = 1
IdleTime = 1800
MaxBuilds = 1
OffPeakIdleCount = 0
OffPeakIdleTime = 1200
MachineDriver = "amazonec2"
MachineName = "gitlab-docker-machine%s"
MachineOptions = [
"amazonec2-access-key=xxxxx",
"amazonec2-secret-key=xxxx",
"amazonec2-region=us-east-1",
"amazonec2-vpc-id=vpc-xxxxx",
"amazonec2-subnet-id=subnet-xxxxx",
"amazonec2-instance-type=t2.micro",
"amazonec2-zone=a"
]