Docker+machine autoscaling: Runtime error
Hi gitlab team ! I'm running into an issue with my current setup:
- Gitlab CE 8.11.0
- Gitlab Runner 1.5.2
- docker-machine version 0.8.1
The runner correctly spawn a machine through docker-machine
, then it fails to talk to the docker daemon (probably firewall issues on my side).
After that, the runner launch a second machine and produce an error.
Here is my config.toml
and journalctl
relevant logs.
concurrent = 4
check_interval = 0
[[runners]]
name = "git-runner-1"
url = "https://______.com/ci"
token = "______"
executor = "docker+machine"
[runners.ssh]
[runners.docker]
tls_verify = false
image = "docker:1.11.2"
privileged = true
disable_cache = false
volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/dev/shm:/dev/shm"]
[runners.cache]
Type = "s3"
ServerAddress = "git-runner-1:9000"
AccessKey = "______"
SecretKey = "______"
BucketName = "runner"
Insecure = true
[runners.machine]
IdleCount = 0
IdleTime = 1800
MaxBuilds = 100
MachineDriver = "google"
MachineName = "ci-%s"
MachineOptions = [
"google-project=______",
"google-zone=europe-west1-c",
"google-machine-image=https://www.googleapis.com/compute/v1/projects/coreos-cloud/global/images/coreos-alpha-1122-0-0-v20160727",
"google-username=core",
"google-machine-type=n1-highcpu-2",
"google-preemptible=true",
"engine-registry-mirror=http://git-runner-1:5000"
]
gitlab-ci-multi-runner[5816]: time="2016-08-24T21:14:46Z" level=info msg="Checking for builds... received" build=477 repo_url="https://____/my/project.git" runner=e51803c6
gitlab-ci-multi-runner[5816]: Running pre-create checks...
gitlab-ci-multi-runner[5816]: (runner-e51803c6-ci-1472073286-c066b248) Check that the project exists
gitlab-ci-multi-runner[5816]: (runner-e51803c6-ci-1472073286-c066b248) Check if the instance already exists
gitlab-ci-multi-runner[5816]: Creating machine...
gitlab-ci-multi-runner[5816]: (runner-e51803c6-ci-1472073286-c066b248) Generating SSH Key
gitlab-ci-multi-runner[5816]: (runner-e51803c6-ci-1472073286-c066b248) Creating host...
gitlab-ci-multi-runner[5816]: (runner-e51803c6-ci-1472073286-c066b248) Opening firewall ports
gitlab-ci-multi-runner[5816]: (runner-e51803c6-ci-1472073286-c066b248) Creating instance
gitlab-ci-multi-runner[5816]: (runner-e51803c6-ci-1472073286-c066b248) Waiting for Instance
gitlab-ci-multi-runner[5816]: (runner-e51803c6-ci-1472073286-c066b248) Uploading SSH Key
gitlab-ci-multi-runner[5816]: Waiting for machine to be running, this may take a few minutes...
gitlab-ci-multi-runner[5816]: Detecting operating system of created instance...
gitlab-ci-multi-runner[5816]: Waiting for SSH to be available...
gitlab-ci-multi-runner[5816]: Detecting the provisioner...
gitlab-ci-multi-runner[5816]: Provisioning with coreOS...
gitlab-ci-multi-runner[5816]: Copying certs to the local machine directory...
gitlab-ci-multi-runner[5816]: Copying certs to the remote machine...
gitlab-ci-multi-runner[5816]: Setting Docker configuration on the remote daemon...
gitlab-ci-multi-runner[5816]: Checking connection to Docker...
gitlab-ci-multi-runner[5816]: Error creating machine: Error checking the host: Error checking and/or regenerating the certs: There was an error validating certificates for host "130.211.78.194:2376": dial tcp 130.211.78.194:2376: i/o timeout
gitlab-ci-multi-runner[5816]: You can attempt to regenerate them using 'docker-machine regenerate-certs [name]'.
gitlab-ci-multi-runner[5816]: Be advised that this will trigger a Docker daemon restart which will stop running containers.
gitlab-ci-multi-runner[5816]: The default lines below are for a sh/bash shell, you can specify the shell you're using, with the --shell flag.
gitlab-runner[5816]: time="2016-08-24T21:15:46Z" level=warning msg="Machine creation failed, trying to provision" error="exit status 1" name=runner-e51803c6-ci-1472073286-c066b248
<nil>
gitlab-ci-multi-runner[5816]: time="2016-08-24T21:15:46Z" level=warning msg="Machine creation failed, trying to provision" error="exit status 1" name=runner-e51803c6-ci-1472073286-c066b248
gitlab-ci-multi-runner[5816]: Waiting for SSH to be available...
gitlab-ci-multi-runner[5816]: Detecting the provisioner...
gitlab-ci-multi-runner[5816]: Copying certs to the local machine directory...
gitlab-ci-multi-runner[5816]: Copying certs to the remote machine...
gitlab-ci-multi-runner[5816]: Setting Docker configuration on the remote daemon...
gitlab-runner[5816]: time="2016-08-24T21:15:54Z" level=info msg="Machine created" name=runner-e51803c6-ci-1472073286-c066b248 now=2016-08-24 21:15:54.670378724 +0000 UTC time=1m8.643059962s
<nil>
gitlab-ci-multi-runner[5816]: time="2016-08-24T21:15:54Z" level=info msg="Machine created" fields.time=1m8.643059962s name=runner-e51803c6-ci-1472073286-c066b248 now=2016-08-24 21:15:54.670378724 +0000 UTC time=1m8.643059962s
gitlab-runner[5816]: time="2016-08-24T21:15:56Z" level=info msg="Build succeeded" build=477 project=1 runner=e51803c6
<nil>
gitlab-ci-multi-runner[5816]: time="2016-08-24T21:15:56Z" level=info msg="Build succeeded" build=477 project=1 runner=e51803c6
gitlab-ci-multi-runner[5816]: panic: runtime error: invalid memory address or nil pointer dereference
gitlab-ci-multi-runner[5816]: [signal 0xb code=0x1 addr=0x10 pc=0x4cdf11]
gitlab-ci-multi-runner[5816]: goroutine 11 [running]:
gitlab-ci-multi-runner[5816]: panic(0xb08700, 0xc82000e060)
gitlab-ci-multi-runner[5816]: /usr/local/go/src/runtime/panic.go:481 +0x3e6
gitlab-ci-multi-runner[5816]: gitlab.com/gitlab-org/gitlab-ci-multi-runner/executors/docker/machine.(*machineProvider).Release(0xc8200112c0, 0xc8200df7d0, 0xb57920, 0xc820302840, 0x0, 0x0)
gitlab-ci-multi-runner[5816]: /go/src/gitlab.com/gitlab-org/gitlab-ci-multi-runner/executors/docker/machine/provider.go:347 +0xd1
gitlab-ci-multi-runner[5816]: gitlab.com/gitlab-org/gitlab-ci-multi-runner/executors/docker/machine.(*machineExecutor).Cleanup(0xc8200df7a0)
gitlab-ci-multi-runner[5816]: /go/src/gitlab.com/gitlab-org/gitlab-ci-multi-runner/executors/docker/machine/executor.go:101 +0xe3
gitlab-ci-multi-runner[5816]: gitlab.com/gitlab-org/gitlab-ci-multi-runner/common.(*Build).retryCreateExecutor(0xc82038e300, 0xc8200471a0, 0x7f24bfde1ff8, 0xc8200112c0, 0x7f24bfdfc268, 0xc820085400, 0xc820456b00, 0x7f24bfdfc300, 0xc8200df7a0, 0x7f24bfddd028, ...)
gitlab-ci-multi-runner[5816]: /go/src/gitlab.com/gitlab-org/gitlab-ci-multi-runner/common/build.go:244 +0x1d7
gitlab-ci-multi-runner[5816]: gitlab.com/gitlab-org/gitlab-ci-multi-runner/common.(*Build).Run(0xc82038e300, 0xc8200471a0, 0x7f24bfdfc268, 0xc820085400, 0x0, 0x0)
gitlab-ci-multi-runner[5816]: /go/src/gitlab.com/gitlab-org/gitlab-ci-multi-runner/common/build.go:284 +0x48a
gitlab-ci-multi-runner[5816]: gitlab.com/gitlab-org/gitlab-ci-multi-runner/commands.(*RunCommand).processRunner(0xc82008a380, 0x0, 0xc82008a540, 0xc820047500, 0x0, 0x0)
gitlab-ci-multi-runner[5816]: /go/src/gitlab.com/gitlab-org/gitlab-ci-multi-runner/commands/multi.go:148 +0xc9b
gitlab-ci-multi-runner[5816]: gitlab.com/gitlab-org/gitlab-ci-multi-runner/commands.(*RunCommand).processRunners(0xc82008a380, 0x0, 0xc8200476e0, 0xc820047500)
gitlab-ci-multi-runner[5816]: /go/src/gitlab.com/gitlab-org/gitlab-ci-multi-runner/commands/multi.go:156 +0x256
gitlab-ci-multi-runner[5816]: created by gitlab.com/gitlab-org/gitlab-ci-multi-runner/commands.(*RunCommand).startWorkers
gitlab-ci-multi-runner[5816]: /go/src/gitlab.com/gitlab-org/gitlab-ci-multi-runner/commands/multi.go:172 +0x96
systemd[1]: gitlab-runner.service: main process exited, code=exited, status=2/INVALIDARGUMENT
systemd[1]: Unit gitlab-runner.service entered failed state.
Thanks for your awesome work guys !
Edited by 🤖 GitLab Bot 🤖