Docker Machine Autoscale Configuration Leaving Orphaned Insances on AWS costing $100s
Summary
I am running CI jobs with the Gitlab Runner docker machine autoscaling configuration on AWS EC2 instances. About 1 every 4 runs, the bastion runner spawns unnamed workers that it then fails to cull. Unfortunately, I failed to notice this was happening for 2 weeks and these orphaned t3.medium instances cost me about $300 dollars in AWS bills.
Interesting pieces of information:
- After investigating the bastion's
/var/log/syslog
output I see the following message each time a EC2 instance is orphaned.
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: Error creating machine: Error in driver during machine creation: Panic in the driver: runtime error: index out of range#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
- I don't know if this is relevant, but everytime orphans are created on AWS they do not have a name. I'm guessing the docker machine driver fails midway through creating them and assumes AWS doesn't actually create the EC2 instance when in fact it does. In the picture above, I had to manually teardown the unnamed instances.
Anyways, I'm a huge fan on Gitlab and have built my entire business on top of its amazing CI capabilities. Unfortunately, though I'm quite scared to use the CI for fear of wasting copious amounts of money.
Any help would be greatly appreciated. Thanks so much for all you guys do.
Steps to reproduce
Unfortunately, this repository is proprietary so it cannot be shared.
Below is the repository's top level gitlab-ci.yml file. The file includes another gitlab-ci.yml file. When executed it kicks off roughly 32 jobs in the first stage.
.gitlab-ci.yml
#
# NAME:
# .gitlab-ci.yml
#
# DESCRIPTION:
# Beblsoft GitLab CI Integration
#
# ENVIRONMENT VARIABLES:
# Following environment variables are defined for Beblsoft Project in GitLab
# - AWS_ACCESS_KEY_ID
# - AWS_SECRET_ACCESS_KEY
#
# ---------------------- GLOBAL STATE --------------------------------------- #
stages:
- Debug
- Build
- TestLocal
- ScheduleDeploy
- ScheduleTest
- StagingDeploy
- StagingTest
- ProductionDeploy
- ProductionTest
variables:
BEBLSOFT: /builds/beblsoft/beblsoft
TOOLS: /builds/beblsoft/beblsoft/tools
BASE: /builds/beblsoft/beblsoft/base
PROJECTS: /builds/beblsoft/beblsoft/projects
SMECKN: /builds/beblsoft/beblsoft/projects/smeckn
QUOTE: /builds/beblsoft/beblsoft/projects/quote
MOJOURNEY: /builds/beblsoft/beblsoft/projects/mojourney
include:
- local: "/projects/smeckn/.gitlab-ci.yml"
# ---------------------- GLOBAL TEST LOCAL ---------------------------------- #
# Test Local Base -----------------
.globalTestLocalBaseJob:
image: registry.gitlab.com/beblsoft/beblsoft/tools/ubuntudind
artifacts:
name: "$CI_COMMIT_REF_NAME-$CI_COMMIT_SHORT_SHA-$CI_JOB_NAME"
untracked: true
when: on_failure
expire_in: 1 week
tags:
- docker
- privileged
stage: TestLocal
# Pylint --------------------------
globalPylintJob:
extends: .globalTestLocalBaseJob
script:
- $TOOLS/pylint/gitlabRunner
# Markdownlint --------------------
globalMarkdownlintJob:
extends: .globalTestLocalBaseJob
script:
- $TOOLS/markdownlint/gitlabRunner
allow_failure: true
Actual behavior
EC2 instances are spawned and then never culled, creating large AWS costs.
Expected behavior
All EC2 instances spawned by the runner should be culled.
Relevant logs and/or screenshots
Output in `/var/log/syslog` from host:
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: Error creating machine: Error in driver during machine creation: Panic in the driver: runtime error: index out of range#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: Error creating machine: Error in driver during machine creation: Panic in the driver: runtime error: index out of range#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: goroutine 58 [running]: #033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: goroutine 58 [running]: #033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: runtime/debug.Stack(0x1534700, 0xc4201bfc00, 0x147d8b8)#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: runtime/debug.Stack(0x1534700, 0xc4201bfc00, 0x147d8b8)#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: #011/usr/local/go/src/runtime/debug/stack.go:24 +0xa7#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: #011/usr/local/go/src/runtime/debug/stack.go:24 +0xa7#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: github.com/docker/machine/libmachine/drivers/rpc.(*StandardStack).Stack(0x1efc880, 0x114ca60, 0x1ec9e00, 0x1534700)#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: github.com/docker/machine/libmachine/drivers/rpc.(*StandardStack).Stack(0x1efc880, 0x114ca60, 0x1ec9e00, 0x1534700)#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: #011/go/src/github.com/docker/machine/libmachine/drivers/rpc/server_driver.go:23 +0x22#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: #011/go/src/github.com/docker/machine/libmachine/drivers/rpc/server_driver.go:23 +0x22#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: github.com/docker/machine/libmachine/drivers/rpc.trapPanic(0xc420513b60)#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: github.com/docker/machine/libmachine/drivers/rpc.trapPanic(0xc420513b60)#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: #011/go/src/github.com/docker/machine/libmachine/drivers/rpc/server_driver.go:129 +0x96#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: #011/go/src/github.com/docker/machine/libmachine/drivers/rpc/server_driver.go:129 +0x96#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: panic(0x114ca60, 0x1ec9e00) #033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: panic(0x114ca60, 0x1ec9e00) #033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: #011/usr/local/go/src/runtime/panic.go:505 +0x229#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: #011/usr/local/go/src/runtime/panic.go:505 +0x229#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: github.com/docker/machine/drivers/amazonec2.(*Driver).getInstance(0xc42019db80, 0x0, 0x0, 0x0)#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: github.com/docker/machine/drivers/amazonec2.(*Driver).getInstance(0xc42019db80, 0x0, 0x0, 0x0)#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: #011/go/src/github.com/docker/machine/drivers/amazonec2/amazonec2.go:928 +0x15b#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: #011/go/src/github.com/docker/machine/drivers/amazonec2/amazonec2.go:928 +0x15b#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: github.com/docker/machine/drivers/amazonec2.(*Driver).GetIP(0xc42019db80, 0x1eca8b0, 0xc420162080, 0xc420092010, 0x0)#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: github.com/docker/machine/drivers/amazonec2.(*Driver).GetIP(0xc42019db80, 0x1eca8b0, 0xc420162080, 0xc420092010, 0x0)#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: #011/go/src/github.com/docker/machine/drivers/amazonec2/amazonec2.go:786 +0x2f#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: #011/go/src/github.com/docker/machine/drivers/amazonec2/amazonec2.go:786 +0x2f#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: github.com/docker/machine/drivers/amazonec2.(*Driver).instanceIpAvailable(0xc42019db80, 0x4c47fe)#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: github.com/docker/machine/drivers/amazonec2.(*Driver).instanceIpAvailable(0xc42019db80, 0x4c47fe)#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: #011/go/src/github.com/docker/machine/drivers/amazonec2/amazonec2.go:530 +0x2f#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: #011/go/src/github.com/docker/machine/drivers/amazonec2/amazonec2.go:530 +0x2f#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: github.com/docker/machine/drivers/amazonec2.(*Driver).(github.com/docker/machine/drivers/amazonec2.instanceIpAvailable)-fm(0x0)#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: github.com/docker/machine/drivers/amazonec2.(*Driver).(github.com/docker/machine/drivers/amazonec2.instanceIpAvailable)-fm(0x0)#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: #011/go/src/github.com/docker/machine/drivers/amazonec2/amazonec2.go:743 +0x2a#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: #011/go/src/github.com/docker/machine/drivers/amazonec2/amazonec2.go:743 +0x2a#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: github.com/docker/machine/libmachine/mcnutils.WaitForSpecific.func1(0xc42055a000, 0xc420556600, 0x2b)#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: github.com/docker/machine/libmachine/mcnutils.WaitForSpecific.func1(0xc42055a000, 0xc420556600, 0x2b)#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: #011/go/src/github.com/docker/machine/libmachine/mcnutils/utils.go:97 +0x26#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: #011/go/src/github.com/docker/machine/libmachine/mcnutils/utils.go:97 +0x26#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: github.com/docker/machine/libmachine/mcnutils.WaitForSpecificOrError(0xc4205137b0, 0x3c, 0xb2d05e00, 0x1, 0x1)#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: github.com/docker/machine/libmachine/mcnutils.WaitForSpecificOrError(0xc4205137b0, 0x3c, 0xb2d05e00, 0x1, 0x1)#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: #011/go/src/github.com/docker/machine/libmachine/mcnutils/utils.go:83 +0x54#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: #011/go/src/github.com/docker/machine/libmachine/mcnutils/utils.go:83 +0x54#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: github.com/docker/machine/libmachine/mcnutils.WaitForSpecific(0xc420513a40, 0x3c, 0xb2d05e00, 0x1, 0x1)#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: github.com/docker/machine/libmachine/mcnutils.WaitForSpecific(0xc420513a40, 0x3c, 0xb2d05e00, 0x1, 0x1)#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: #011/go/src/github.com/docker/machine/libmachine/mcnutils/utils.go:96 +0x55#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: #011/go/src/github.com/docker/machine/libmachine/mcnutils/utils.go:96 +0x55#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: github.com/docker/machine/libmachine/mcnutils.WaitFor(0xc420513a40, 0x1, 0x1)#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: github.com/docker/machine/libmachine/mcnutils.WaitFor(0xc420513a40, 0x1, 0x1)#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: #011/go/src/github.com/docker/machine/libmachine/mcnutils/utils.go:102 +0x3e#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: #011/go/src/github.com/docker/machine/libmachine/mcnutils/utils.go:102 +0x3e#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: github.com/docker/machine/drivers/amazonec2.(*Driver).innerCreate(0xc42019db80, 0x0, 0x0)#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: github.com/docker/machine/drivers/amazonec2.(*Driver).innerCreate(0xc42019db80, 0x0, 0x0)#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: #011/go/src/github.com/docker/machine/drivers/amazonec2/amazonec2.go:743 +0x11be#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: #011/go/src/github.com/docker/machine/drivers/amazonec2/amazonec2.go:743 +0x11be#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: github.com/docker/machine/drivers/amazonec2.(*Driver).Create(0xc42019db80, 0x147d768, 0xc420065b60)#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: #011/go/src/github.com/docker/machine/drivers/amazonec2/amazonec2.go:585 +0x4c#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: github.com/docker/machine/libmachine/drivers/rpc.(*RPCServerDriver).Create(0xc42000a980, 0x1efcc38, 0x1efcc38, 0x0, 0x0)#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: github.com/docker/machine/drivers/amazonec2.(*Driver).Create(0xc42019db80, 0x147d768, 0xc420065b60)#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: #011/go/src/github.com/docker/machine/libmachine/drivers/rpc/server_driver.go:140 +0x65#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: reflect.Value.call(0xc420076840, 0xc42000c2f0, 0x13, 0x141760a, 0x4, 0xc420065f18, 0x3, 0x3, 0xc420073100, 0x0, ...)#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: #011/usr/local/go/src/reflect/value.go:447 +0x969#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: #011/go/src/github.com/docker/machine/drivers/amazonec2/amazonec2.go:585 +0x4c#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: reflect.Value.Call(0xc420076840, 0xc42000c2f0, 0x13, 0xc42004e718, 0x3, 0x3, 0xc420080600, 0xc4200a07e0, 0xc42004e798)#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: #011/usr/local/go/src/reflect/value.go:308 +0xa4#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: net/rpc.(*service).call(0xc420072ec0, 0xc42017c230, 0xc420032550, 0xc420032560, 0xc420030e80, 0xc42000afc0, 0x104fea0, 0x1efcc38, 0x16, 0x104fea0, ...)#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: #011/usr/local/go/src/net/rpc/server.go:384 +0x14e#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: github.com/docker/machine/libmachine/drivers/rpc.(*RPCServerDriver).Create(0xc42000a980, 0x1efcc38, 0x1efcc38, 0x0, 0x0)#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: created by net/rpc.(*Server).ServeCodec #033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: #011/usr/local/go/src/net/rpc/server.go:480 +0x43a#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: #011/go/src/github.com/docker/machine/libmachine/drivers/rpc/server_driver.go:140 +0x65#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: reflect.Value.call(0xc420076840, 0xc42000c2f0, 0x13, 0x141760a, 0x4, 0xc420065f18, 0x3, 0x3, 0xc420073100, 0x0, ...)#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: #011/usr/local/go/src/reflect/value.go:447 +0x969#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: reflect.Value.Call(0xc420076840, 0xc42000c2f0, 0x13, 0xc42004e718, 0x3, 0x3, 0xc420080600, 0xc4200a07e0, 0xc42004e798)#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: #011/usr/local/go/src/reflect/value.go:308 +0xa4#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: net/rpc.(*service).call(0xc420072ec0, 0xc42017c230, 0xc420032550, 0xc420032560, 0xc420030e80, 0xc42000afc0, 0x104fea0, 0x1efcc38, 0x16, 0x104fea0, ...)#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: #011/usr/local/go/src/net/rpc/server.go:384 +0x14e#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: created by net/rpc.(*Server).ServeCodec #033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Aug 16 18:09:52 ip-10-0-0-44 gitlab-runner[5828]: #033[31;1mERROR: #011/usr/local/go/src/net/rpc/server.go:480 +0x43a#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-81qzx4fj-1565978986-b75d4a7d #033[31;1moperation#033[0;m=create
Environment description
Using Gitlab Runner docker-machine autoscaling configuration on AWS.
The bastion instance has the following docker-machine and docker versions.
ubuntu@ip-10-0-0-44:~$ docker-machine -v
docker-machine version 0.16.0, build 702c267f
ubuntu@ip-10-0-0-44:~$ docker -v
Docker version 19.03.1, build 74b1e89
config.toml contents
# Global Options
concurrent = 80 # Total number of jobs for whole file, tune to bastion size
check_interval = 0
log_level = "debug"
[session_server]
session_timeout = 1800 # How long in seconds the session can stay active after the
# job completes (which will block job from finishing)
# Docker Machine Runner
[[runners]]
name = "AWS-Docker-Machine"
url = "https://gitlab.com/"
token = "X"
executor = "docker+machine"
limit = 50
request_concurrency = 1
environment = ["DOCKER_DRIVER=overlay2"]
[runners.docker]
tls_verify = false
image = "ubuntu:bionic"
privileged = true
disable_cache = true
volumes = []
pull_policy = "always"
[runners.cache]
Type = "s3"
Path = ""
Shared = true
[runners.cache.s3]
ServerAddress = "s3.amazonaws.com"
AccessKey = "X"
SecretKey = "X"
BucketName = "X"
BucketLocation = "us-east-1"
Insecure = false
[runners.machine]
IdleCount = 0
IdleTime = 3
MaxBuilds = 10
MachineDriver = "amazonec2"
MachineName = "%s"
MachineOptions = [
"amazonec2-access-key=X",
"amazonec2-secret-key=X",
"amazonec2-region=us-east-1",
"amazonec2-vpc-id=X",
"amazonec2-subnet-id=X",
"amazonec2-zone=a",
"amazonec2-instance-type=t3.medium",
"amazonec2-tags=runner-manager-name,gitlab-aws-autoscaler,gitlab,true,gitlab-runner-autoscale,true"]
OffPeakTimezone = ""
OffPeakIdleCount = 0
OffPeakIdleTime = 0
Used GitLab Runner version
ubuntu@ip-10-0-0-44:~$ sudo gitlab-runner --version
Version: 12.1.0
Git revision: de7731dd
Git branch: 12-1-stable
GO version: go1.8.7
Built: 2019-07-19T13:53:04+0000
OS/Arch: linux/amd64
Possible fixes
The stack trace points to the amazon ec2 create code path.
I need to learn go and start peeling back the onion :).