Gitlab runner fails to launch ec2 instance: InvalidBlockDeviceMapping Volume of size 16GB is smaller than snapshot
Summary
Gitlab runner fails to launch ec2 instance from ami with 50GB volume.
Steps to reproduce
- Create custom ami with volume bigger than 16GB
- Configure
amazonec2-ami=
to point to the ami from step 1 - Run a job on the runner manager configured with custom ami from step 2
- Check logs and see the error:
level=error msg="Error creating machine: Error in driver during machine creation: Error request spot instance: InvalidBlockDeviceMapping: Volume of size 16GB is smaller than snapshot 'snap-xxxx', expect size"
.gitlab-ci.yml
stages:
- deploy
default:
tags:
- "common"
variables:
ENVIRONMENT:
value: "staging"
description: "Choose which environment to deploy to. Valid values: prod/staging"
deploy-staging:
stage: deploy
variables:
ENVIRONMENT: "staging"
rules:
- if: '$CI_COMMIT_BRANCH == "staging"'
script:
- bash scripts/deploy.sh "${ENVIRONMENT}"
deploy-prod:
stage: deploy
variables:
ENVIRONMENT: "prod"
rules:
- if: '$CI_COMMIT_BRANCH == "master"'
script:
- bash scripts/deploy.sh "${ENVIRONMENT}"
deploy-manual:
stage: deploy
rules:
- when: manual
allow_failure: true # make this job none blocking
script:
- bash scripts/deploy.sh "${ENVIRONMENT}"
Actual behavior
Gitlab runner manager fails to launch ec2 instance.
Expected behavior
Gitlab runner should have launched an ec2 instance based on the ami.
Relevant logs and/or screenshots
job log
Mar 15 13:34:56 ip-172-31-86-111 gitlab-runner[150117]: time="2022-03-15T13:34:56Z" level=info msg="Running pre-create checks..." driver=amazonec2 name=runner-y6ah9w9s-gitlab-runner-1647351296-4f76219b operation=create
Mar 15 13:34:56 ip-172-31-86-111 gitlab-runner[150117]: time="2022-03-15T13:34:56Z" level=info msg="Creating machine..." driver=amazonec2 name=runner-y6ah9w9s-gitlab-runner-1647351296-4f76219b operation=create
Mar 15 13:34:56 ip-172-31-86-111 gitlab-runner[150117]: time="2022-03-15T13:34:56Z" level=info msg="(runner-y6ah9w9s-gitlab-runner-1647351296-4f76219b) Launching instance..." driver=amazonec2 name=runner-y6ah9w9s-gitlab-runner-1647351296-4f76219b operation=create
Mar 15 13:34:56 ip-172-31-86-111 gitlab-runner[150117]: time="2022-03-15T13:34:56Z" level=info msg="IdleCount is set to 0 so the machine will be created on demand in job context" creating=1 idle=0 idleCount=0 idleCountMin=0 idleScaleFactor=0 maxMachineCreate=0 maxMachines=0 removing=0 runner=y6AH9w9S total=1 used=0
Mar 15 13:34:57 ip-172-31-86-111 gitlab-runner[150117]: time="2022-03-15T13:34:57Z" level=info msg="(runner-y6ah9w9s-gitlab-runner-1647351296-4f76219b) Missing instance ID, this is likely due to a failure during machine creation" driver=amazonec2 name=runner-y6ah9w9s-gitlab-runner-1647351296-4f76219b operation=create
Mar 15 13:34:59 ip-172-31-86-111 gitlab-runner[150117]: time="2022-03-15T13:34:59Z" level=info msg="IdleCount is set to 0 so the machine will be created on demand in job context" creating=1 idle=0 idleCount=0 idleCountMin=0 idleScaleFactor=0 maxMachineCreate=0 maxMachines=0 removing=0 runner=y6AH9w9S total=1 used=0
Mar 15 13:34:59 ip-172-31-86-111 gitlab-runner[150117]: time="2022-03-15T13:34:59Z" level=error msg="Error creating machine: Error in driver during machine creation: Error request spot instance: InvalidBlockDeviceMapping: Volume of size 16GB is smaller than snapshot 'snap-03c789bb4514892c9', expect size" driver=amazonec2 name=runner-y6ah9w9s-gitlab-runner-1647351296-4f76219b operation=create
Mar 15 13:34:59 ip-172-31-86-111 gitlab-runner[150117]: time="2022-03-15T13:34:59Z" level=error msg="\tstatus code: 400, request id: 49ea622e-8cce-47f0-9e07-3b77f3817675" driver=amazonec2 name=runner-y6ah9w9s-gitlab-runner-1647351296-4f76219b operation=create
Mar 15 13:34:59 ip-172-31-86-111 gitlab-runner[150117]: time="2022-03-15T13:34:59Z" level=warning msg="Problem while reading command output" error="read |0: file already closed"
Mar 15 13:34:59 ip-172-31-86-111 gitlab-runner[150117]: time="2022-03-15T13:34:59Z" level=warning msg="Problem while reading command output" error="read |0: file already closed"
Mar 15 13:34:59 ip-172-31-86-111 gitlab-runner[150117]: time="2022-03-15T13:34:59Z" level=error msg="Machine creation failed" error="exit status 1" fields.time=3.70815809s name=runner-y6ah9w9s-gitlab-runner-1647351296-4f76219b
Mar 15 13:34:59 ip-172-31-86-111 gitlab-runner[150117]: time="2022-03-15T13:34:59Z" level=warning msg="Requesting machine removal" lifetime=3.708983075s name=runner-y6ah9w9s-gitlab-runner-1647351296-4f76219b now="2022-03-15 13:34:59.837903442 +0000 UTC m=+105.427307392" reason="Failed to create" used=3.708984087s usedCount=0
Mar 15 13:34:59 ip-172-31-86-111 gitlab-runner[150117]: time="2022-03-15T13:34:59Z" level=warning msg="Stopping machine" lifetime=3.729928316s name=runner-y6ah9w9s-gitlab-runner-1647351296-4f76219b reason="Failed to create" used=20.917641ms usedCount=0
Mar 15 13:34:59 ip-172-31-86-111 gitlab-runner[150117]: time="2022-03-15T13:34:59Z" level=info msg="Stopping \"runner-y6ah9w9s-gitlab-runner-1647351296-4f76219b\"..." name=runner-y6ah9w9s-gitlab-runner-1647351296-4f76219b operation=stop
Mar 15 13:35:00 ip-172-31-86-111 gitlab-runner[150117]: time="2022-03-15T13:35:00Z" level=error msg="InvalidInstanceID.Malformed: Invalid id: \"\"" name=runner-y6ah9w9s-gitlab-runner-1647351296-4f76219b operation=stop
Mar 15 13:35:00 ip-172-31-86-111 gitlab-runner[150117]: time="2022-03-15T13:35:00Z" level=error msg="\tstatus code: 400, request id: b30348de-3518-419b-b7b0-b36e71181316" name=runner-y6ah9w9s-gitlab-runner-1647351296-4f76219b operation=stop
Mar 15 13:35:00 ip-172-31-86-111 gitlab-runner[150117]: time="2022-03-15T13:35:00Z" level=warning msg="Problem while reading command output" error="read |0: file already closed"
Mar 15 13:35:00 ip-172-31-86-111 gitlab-runner[150117]: time="2022-03-15T13:35:00Z" level=warning msg="Problem while reading command output" error="read |0: file already closed"
Mar 15 13:35:00 ip-172-31-86-111 gitlab-runner[150117]: time="2022-03-15T13:35:00Z" level=warning msg="Error while stopping machine" error="exit status 1" lifetime=3.9001056s name=runner-y6ah9w9s-gitlab-runner-1647351296-4f76219b reason="Failed to create" used=191.094682ms usedCount=0
Mar 15 13:35:00 ip-172-31-86-111 gitlab-runner[150117]: time="2022-03-15T13:35:00Z" level=warning msg="Removing machine" lifetime=3.901213823s name=runner-y6ah9w9s-gitlab-runner-1647351296-4f76219b reason="Failed to create" used=192.2027ms usedCount=0
Mar 15 13:35:00 ip-172-31-86-111 gitlab-runner[150117]: time="2022-03-15T13:35:00Z" level=info msg="About to remove runner-y6ah9w9s-gitlab-runner-1647351296-4f76219b" name=runner-y6ah9w9s-gitlab-runner-1647351296-4f76219b operation=remove
Mar 15 13:35:00 ip-172-31-86-111 gitlab-runner[150117]: time="2022-03-15T13:35:00Z" level=info msg="WARNING: This action will delete both local reference and remote instance." name=runner-y6ah9w9s-gitlab-runner-1647351296-4f76219b operation=remove
Mar 15 13:35:00 ip-172-31-86-111 gitlab-runner[150117]: time="2022-03-15T13:35:00Z" level=info msg="(runner-y6ah9w9s-gitlab-runner-1647351296-4f76219b) Missing instance ID, this is likely due to a failure during machine creation" name=runner-y6ah9w9s-gitlab-runner-1647351296-4f76219b operation=remove
Mar 15 13:35:00 ip-172-31-86-111 gitlab-runner[150117]: time="2022-03-15T13:35:00Z" level=info msg="Successfully removed runner-y6ah9w9s-gitlab-runner-1647351296-4f76219b" name=runner-y6ah9w9s-gitlab-runner-1647351296-4f76219b operation=remove
Mar 15 13:35:00 ip-172-31-86-111 gitlab-runner[150117]: time="2022-03-15T13:35:00Z" level=warning msg="Problem while reading command output" error="read |0: file already closed"
Mar 15 13:35:00 ip-172-31-86-111 gitlab-runner[150117]: time="2022-03-15T13:35:00Z" level=warning msg="Problem while reading command output" error="read |0: file already closed"
Mar 15 13:35:00 ip-172-31-86-111 gitlab-runner[150117]: time="2022-03-15T13:35:00Z" level=info msg="Machine removed" lifetime=4.122219589s name=runner-y6ah9w9s-gitlab-runner-1647351296-4f76219b now="2022-03-15 13:35:00.251140747 +0000 UTC m=+105.840544718" reason="Failed to create" retries=0 used=413.208517ms usedCount=0
Mar 15 13:35:00 ip-172-31-86-111 gitlab-runner[150117]: time="2022-03-15T13:35:00Z" level=info msg="Cleaned up docker-machine" job=2205636423 project=6961934 runner=y6AH9w9S
Mar 15 13:35:00 ip-172-31-86-111 gitlab-runner[150117]: time="2022-03-15T13:35:00Z" level=warning msg="Preparation failed: exit status 1" job=2205636423 project=6961934 runner=y6AH9w9S
Mar 15 13:35:00 ip-172-31-86-111 gitlab-runner[150117]: time="2022-03-15T13:35:00Z" level=info msg="Will be retried in 3s ..." job=2205636423 project=6961934 runner=y6AH9w9S
Mar 15 13:35:02 ip-172-31-86-111 gitlab-runner[150117]: time="2022-03-15T13:35:02Z" level=info msg="IdleCount is set to 0 so the machine will be created on demand in job context" creating=0 idle=0 idleCount=0 idleCountMin=0 idleScaleFactor=0 maxMachineCreate=0 maxMachines=0 removing=0 runner=y6AH9w9S total=0 used=0
Environment description
- Custom gitlab runners
- Gitlab runner manager is on ubuntu 20.04 LTS
- Docker version 20.10.13, build a224086
- docker-machine version 0.16.2-gitlab.11, build b09ad09f
- Gitlab runner is registered on a Group level
config.toml contents
concurrent = 5
check_interval = 3
log_level = "info"
log_format = "text"
[session_server]
listen_address = "[::]:8093"
advertise_address = "runner-host-name.tld:8093"
session_timeout = 1800
[[runners]]
name = "gitlab-common-runner"
url = "https://gitlab.com/"
token = "<token>"
executor = "docker+machine"
[runners.cache]
Type = "s3"
Shared = true
Path = "cicd/runners-cache/"
[runners.cache.s3]
AuthenticationType = "iam"
BucketName = "our-bucket"
Insecure = false
[runners.docker]
tls_verify = false
image = "python:3.10.2-slim-buster"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = true
volumes = ["/cache"]
shm_size = 0
[runners.machine]
IdleCount = 0
IdleScaleFactor = 0.0
IdleCountMin = 0
MaxBuilds = 10
MachineDriver = "amazonec2"
MachineName = "gitlab-runner-%s"
MachineOptions = [
"amazonec2-region=us-east-1",
"amazonec2-vpc-id=vpc-xxxx",
"amazonec2-subnet-id=subnet-xxxxx",
"amazonec2-zone=c",
"amazonec2-use-private-address=true",
"amazonec2-tags=gitlab-runner,common",
"amazonec2-security-group=gitlab-runner-manager",
"amazonec2-instance-type=t3.xlarge",
"amazonec2-ssh-user=ubuntu",
"amazonec2-iam-instance-profile=gitlab-runner",
"amazonec2-ami=ami-xxxx",
"amazonec2-request-spot-instance=true",
"amazonec2-spot-price=0.05",
"amazonec2-block-duration-minutes=60",
]
Used GitLab Runner version
Possible fixes
Build ami with 16GB volume.
Edited by Ana Trakhtman