Upgraded to v0.16.2-gitlab.19 and failure to provision runner instances using docker+machine with spot instances
Was on version, v0.16.2-gitlab.11
Experienced the same certificate issue mentioned in #99.
Feb 3 02:41:44 ip-10-156-200-21 gitlab-runner[60301]: #033[31;1mERROR: ' | sudo tee /etc/docker/ca.pem
...
Feb 3 02:41:44 ip-10-156-200-21 gitlab-runner[60301]: #033[31;1mERROR: tee: /etc/docker/ca.pem: No such file or directory#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-qsbse5ad-docker-linux-16753
91995-eaeb01f4 #033[31;1moperation#033[0;m=create
Feb 3 02:41:44 ip-10-156-200-21 gitlab-runner[60301]: #033[31;1mERROR: Machine creation failed #033[0;m #033[31;1merror#033[0;m=exit status 1 #033[31;1mname#033[0;m=runner-qsbse5ad-docker-linux-1675391995-eaeb01f4 #033[31;1mtime#033[0;m=1m48.413317213s
Also noticed an issue with apt having trouble with focal-backports
GPG key.
Specifically,
ERROR: W: GPG error: http://archive.ubuntu.com/ubuntu focal InRelease: Splitting up /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_focal_InRelease into data and signature failed
Full log:
eb 3 02:41:37 ip-10-156-200-21 gitlab-runner[60301]: #033[31;1mERROR: Get:34 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [16.3 kB]#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-eervmgqj-docker-linux-1675392057-5e8984c7 #033[31;1moperation#033[0;m=create
Feb 3 02:41:37 ip-10-156-200-21 gitlab-runner[60301]: #033[31;1mERROR: Get:35 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B]#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-eervmgqj-docker-linux-1675392057-5e8984c7 #033[31;1moperation#033[0;m=create
Feb 3 02:41:37 ip-10-156-200-21 gitlab-runner[60301]: #033[31;1mERROR: Get:36 http://archive.ubuntu.com/ubuntu focal-backports/multiverse amd64 c-n-f Metadata [116 B]#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-eervmgqj-docker-linux-1675392057-5e8984c7 #033[31;1moperation#033[0;m=create
Feb 3 02:41:37 ip-10-156-200-21 gitlab-runner[60301]: #033[31;1mERROR: Reading package lists... #033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-eervmgqj-docker-linux-1675392057-5e8984c7 #033[31;1moperation#033[0;m=create
Feb 3 02:41:37 ip-10-156-200-21 gitlab-runner[60301]: #033[31;1mERROR: W: GPG error: http://archive.ubuntu.com/ubuntu focal InRelease: Splitting up /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_focal_InRelease into data and signature failed#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-eervmgqj-docker-linux-1675392057-5e8984c7 #033[31;1moperation#033[0;m=create
Feb 3 02:41:37 ip-10-156-200-21 gitlab-runner[60301]: #033[31;1mERROR: E: The repository 'http://archive.ubuntu.com/ubuntu focal InRelease' is not signed.#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-eervmgqj-docker-linux-1675392057-5e8984c7 #033[31;1moperation#033[0;m=create
Feb 3 02:41:37 ip-10-156-200-21 gitlab-runner[60301]: #033[0;33mWARNING: Problem while reading command output #033[0;m #033[0;33merror#033[0;m=read |0: file already closed
Feb 3 02:41:37 ip-10-156-200-21 gitlab-runner[60301]: #033[0;33mWARNING: Problem while reading command output #033[0;m #033[0;33merror#033[0;m=read |0: file already closed
Feb 3 02:41:37 ip-10-156-200-21 gitlab-runner[60301]: #033[31;1mERROR: Machine creation failed #033[0;m #033[31;1merror#033[0;m=exit status 1 #033[31;1mname#033[0;m=runner-eervmgqj-docker
Using v0.16.2-gitlab.11
the instances actually get provisioned. But, because of the certificate and apt
error. It fails.
Upgrade to v0.16.2-gitlab.19
Summary of new issue Machine creation failed
. Logs aren't super helpful as to what exactly the issue is. But, complains about
ERROR: Error creating machine: Error in driver during machine creation: Error request spot instance: InvalidParameter: 'instance' is not a valid taggable resource type for this operation.
Found this issue: gitlab-org/gitlab-runner#29213 (moved)
Full log:
Feb 3 01:41:02 ip-10-156-200-35 gitlab-runner[82162]: #033[31;1mERROR: Machine creation failed #033[0;m #033[31;1merror#033[0;m=exit status 1 #033[31;1mname#033[0;m=runner-6zgjyi7s-docker-linux-1675388458-a372bfab #033[31;1mtime#033[0;m=3.383110228s
Feb 3 01:41:02 ip-10-156-200-35 gitlab-runner[82162]: #033[31;1mERROR: Error creating machine: Error in driver during machine creation: Error request spot instance: InvalidParameter: 'instance' is not a valid taggable resource type for this operation.#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-aprcrwx-docker-linux-1675388459-bfe3a7c4 #033[31;1moperation#033[0;m=create
Feb 3 01:41:02 ip-10-156-200-35 gitlab-runner[82162]: #033[31;1mERROR: #011status code: 400, request id: e70fd07f-eeaf-4b7c-95ab-531f44d4977c#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-aprcrwx-docker-linux-1675388459-bfe3a7c4 #033[31;1moperation#033[0;m=create
Feb 3 01:41:02 ip-10-156-200-35 gitlab-runner[82162]: #033[31;1mERROR: Machine creation failed #033[0;m #033[31;1merror#033[0;m=exit status 1 #033[31;1mname#033[0;m=runner-aprcrwx-docker-linux-1675388459-bfe3a7c4 #033[31;1mtime#033[0;m=3.261242386s
Feb 3 01:41:02 ip-10-156-200-35 gitlab-runner[82162]: #033[31;1mERROR: InvalidInstanceID.Malformed: The instance ID '' is malformed#033[0;m #033[31;1mname#033[0;m=runner-6zgjyi7s-docker-linux-1675388458-a372bfab #033[31;1moperation#033[0;m=stop
Feb 3 01:41:02 ip-10-156-200-35 gitlab-runner[82162]: #033[31;1mERROR: #011status code: 400, request id: 11c9aa81-82f5-409b-acdd-75c117f55da0#033[0;m #033[31;1mname#033[0;m=runner-6zgjyi7s-docker-linux-1675388458-a372bfab #033[31;1moperation#033[0;m=stop
Feb 3 01:41:02 ip-10-156-200-35 gitlab-runner[82162]: #033[31;1mERROR: InvalidInstanceID.Malformed: The instance ID '' is malformed#033[0;m #033[31;1mname#033[0;m=runner-aprcrwx-docker-linux-1675388459-bfe3a7c4 #033[31;1moperation#033[0;m=stop
Feb 3 01:41:02 ip-10-156-200-35 gitlab-runner[82162]: #033[31;1mERROR: #011status code: 400, request id: 1b00b1f4-1259-4387-907d-84f9645eefd3#033[0;m #033[31;1mname#033[0;m=runner-aprcrwx-docker-linux-1675388459-bfe3a7c4 #033[31;1moperation#033[0;m=stop
Feb 3 01:41:03 ip-10-156-200-35 gitlab-runner[82162]: #033[31;1mERROR: Error creating machine: Error in driver during machine creation: Error request spot instance: InvalidParameter: 'instance' is not a valid taggable resource type for this operation.#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-aprcrwx-docker-linux-1675388459-083b3c36 #033[31;1moperation#033[0;m=create
Feb 3 01:41:03 ip-10-156-200-35 gitlab-runner[82162]: #033[31;1mERROR: #011status code: 400, request id: c10bd023-288c-476f-b7be-45f2a4772135#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-aprcrwx-docker-linux-1675388459-083b3c36 #033[31;1moperation#033[0;m=create
Feb 3 01:41:03 ip-10-156-200-35 gitlab-runner[82162]: #033[31;1mERROR: Machine creation failed #033[0;m #033[31;1merror#033[0;m=exit status 1 #033[31;1mname#033[0;m=runner-aprcrwx-docker-linux-1675388459-083b3c36 #033[31;1mtime#033[0;m=4.24396887s
Feb 3 01:41:03 ip-10-156-200-35 gitlab-runner[82162]: #033[31;1mERROR: Error creating machine: Error in driver during machine creation: Error request spot instance: InvalidParameter: 'instance' is not a valid taggable resource type for this operation.#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-fupf4r65-docker-linux-1675388460-ecf92493 #033[31;1moperation#033[0;m=create
Feb 3 01:41:03 ip-10-156-200-35 gitlab-runner[82162]: #033[31;1mERROR: #011status code: 400, request id: 95e40214-1f64-408f-bb9d-37445d00bfe9#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-fupf4r65-docker-linux-1675388460-ecf92493 #033[31;1moperation#033[0;m=create
Feb 3 01:41:03 ip-10-156-200-35 gitlab-runner[82162]: #033[31;1mERROR: Machine creation failed #033[0;m #033[31;1merror#033[0;m=exit status 1 #033[31;1mname#033[0;m=runner-fupf4r65-docker-linux-1675388460-ecf92493 #033[31;1mtime#033[0;m=3.209870542s
Feb 3 01:41:03 ip-10-156-200-35 gitlab-runner[82162]: #033[31;1mERROR: Error creating machine: Error in driver during machine creation: Error request spot instance: InvalidParameter: 'instance' is not a valid taggable resource type for this operation.#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-fupf4r65-docker-linux-1675388460-c040c3ef #033[31;1moperation#033[0;m=create
Feb 3 01:41:03 ip-10-156-200-35 gitlab-runner[82162]: #033[31;1mERROR: #011status code: 400, request id: d680f875-5524-4b90-a217-d61679ae31db#033[0;m #033[31;1mdriver#033[0;m=amazonec2 #033[31;1mname#033[0;m=runner-fupf4r65-docker-linux-1675388460-c040c3ef #033[31;1moperation#033[0;m=create
Feb 3 01:41:03 ip-10-156-200-35 gitlab-runner[82162]: #033[31;1mERROR: Machine creation failed #033[0;m #033[31;1merror#033[0;m=exit status 1 #033[31;1mname#033[0;m=runner-fupf4r65-docker-linux-1675388460-c040c3ef #033[31;1mtime#033[0;m=3.301820324s
Feb 3 01:41:03 ip-10-156-200-35 gitlab-runner[82162]: #033[31;1mERROR: InvalidInstanceID.Malformed: The instance ID '' is malformed#033[0;m #033[31;1mname#033[0;m=runner-aprcrwx-docker-linux-1675388459-083b3c36 #033[31;1moperation#033[0;m=stop
Feb 3 01:41:03 ip-10-156-200-35 gitlab-runner[82162]: #033[31;1mERROR: #011status code: 400, request id: acf82fa8-551c-42d3-aac1-0ae1b924f4be#033[0;m #033[31;1mname#033[0;m=runner-aprcrwx-docker-linux-1675388459-083b3c36 #033[31;1moperation#033[0;m=stop
Feb 3 01:41:03 ip-10-156-200-35 gitlab-runner[82162]: #033[31;1mERROR: InvalidInstanceID.Malformed: The instance ID '' is malformed#033[0;m #033[31;1mname#033[0;m=runner-fupf4r65-docker-linux-1675388460-ecf92493 #033[31;1moperation#033[0;m=stop
Feb 3 01:41:03 ip-10-156-200-35 gitlab-runner[82162]: #033[31;1mERROR: #011status code: 400, request id: 79328c53-02f5-4932-a9f5-09e34f956b07#033[0;m #033[31;1mname#033[0;m=runner-fupf4r65-docker-linux-1675388460-ecf92493 #033[31;1moperation#033[0;m=stop
Feb 3 01:41:03 ip-10-156-200-35 gitlab-runner[82162]: #033[31;1mERROR: InvalidInstanceID.Malformed: The instance ID '' is malformed#033[0;m #033[31;1mname#033[0;m=runner-fupf4r65-docker-linux-1675388460-c040c3ef #033[31;1moperation#033[0;m=stop
Feb 3 01:41:03 ip-10-156-200-35 gitlab-runner[82162]: #033[31;1mERROR: #011status code: 400, request id: 155de243-142b-4ead-8b05-9c4601e5d651#033[0;m #033[31;1mname#033[0;m=runner-fupf4r65-docker-linux-1675388460-c040c3ef #033[31;1moperation#033[0;m=stop
Using Ubuntu 20.04LTS AMI's found here: https://cloud-images.ubuntu.com/locator/ec2/
Running out of AWS us-west-2
Tried AMI listed here to see if it would help, to no avail: !53 (diffs)
Nothing else infrastructure wise has changed (vpc, security groups, etc.).
Runner config.toml
[[runners]]
name = "account-name docker linux build autoscaler 2c-1gb"
url = "https://gitlab.com/"
id = 20873420
token = "XXXXXX"
token_obtained_at = 2023-02-03T02:36:53Z
token_expires_at = 0001-01-01T00:00:00Z
executor = "docker+machine"
environment = ["DOCKER_TLS_CERTDIR="]
[runners.custom_build_dir]
[runners.cache]
Type = "s3"
Shared = true
MaxUploadedArchiveSize = 0
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
image = "alpine:latest"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = true
shm_size = 0
[runners.machine]
IdleCount = 0
IdleScaleFactor = 0.0
IdleCountMin = 0
IdleTime = 900
MaxBuilds = 30
MachineDriver = "amazonec2"
MachineName = "docker-linux-%s"
MachineOptions = ["amazonec2-ami=ami-0e6dff8bde9a09539", "amazonec2-region=us-west-2", "amazonec2-vpc-id=vpc-XXX", "amazonec2-subnet-id=subnet-XXX", "amazonec2-zone=b", "amazonec2-tags=runner-manager-name,gitlab-aws-autoscaler,gitlab,true,gitlab-runner-autoscale,true", "amazonec2-security-group=account-name-build-infrastructure-sg", "amazonec2-instance-type=t3a.micro", "amazonec2-private-address-only=true", "amazonec2-iam-instance-profile=account-name-build-infrastructure-role", "amazonec2-request-spot-instance=true", "amazonec2-spot-price=", "amazonec2-root-size=25"]
[[runners.machine.autoscaling]]
Periods = ["* * 9-20 * * mon-fri *"]
Timezone = "Local"
IdleCount = 2
IdleScaleFactor = 1.5
IdleCountMin = 2
IdleTime = 3600
[[runners.machine.autoscaling]]
Periods = ["* * * * * sat,sun *"]
Timezone = "Local"
IdleCount = 0
IdleScaleFactor = 1.5
IdleCountMin = 2
IdleTime = 3600
Related Issues:
Semi-working solution:
Stop using spots and use on-demand instances with Gitlab v0.16.2-gitlab.19
. Be cool if we could have spots capability back and see a fix for that GPG issue (Ubuntu related). Seems the CKI fork of a fork has the capability for spots. But, would prefer to use Gitlab supported codebase.
I imagine this issue has been painful for all involved. You have my sympathy. Thanks.