docker+machine gitlab executor with private ECR repo: `no basic auth credentials`
Summary
I am trying to setup an environment where I have docker+machine spawning runners, where the images in .gitlab-ci.yml
are stored in ECR.
After reading this - https://docs.gitlab.com/ce/ci/docker/using_docker_images.html#using-credential-helpers, I built an ami containing docker-credential-ecr-login
in path /usr/bin/local
The problem I have now is ensuring that the dockerd
on that machine is able to utilise this. Note that I am relying on the role assumed by the ec2 that is spawned to contain the authorisation to read from ECR.
I am able to ssh into the machine created by gitlab-runner, and run docker pull 1234567890.dkr.ecr.ap-southeast-2.amazonaws.com/my-ci-image
when I configure the following things:
-
export AWS_SDK_LOAD_CONFIG=true
- documented by AWS as being required if using an assumed role
If you are working with an assumed role please set the environment variable: AWS_SDK_LOAD_CONFIG=true also.
- Set
~/.docker/config.json
on the spawned instance to contain :
{
"credsStore": "ecr-login"
}
So that validates that docker-credential-ecr-login
is present and works.
However I am having no luck getting the spawned runner to pull the image when running the job. I have tried a few different ways to specify DOCKER_AUTH_CONFIG
, including as a variable, and as a --env
option during gitlab-runner register. I see:
Running with gitlab-runner 12.0.1 (0e5417a3)
on <my gitlab bastion host> KKhxk4ma
Using Docker executor with image 1234567890.dkr.ecr.ap-southeast-2.amazonaws.com/my-ci:latest ...
WARNING: Container based cache volumes creation is disabled. Will not create volume for "/cache"
Pulling docker image 1234567890.dkr.ecr.ap-southeast-2.amazonaws.com/my-ci:latest ...
ERROR: Preparation failed: Error response from daemon: Get https://1234567890.dkr.ecr.ap-southeast-2.amazonaws.com/v2/my-ci/manifests/latest: no basic auth credentials (executor_docker.go:175:0s)
Will be retried in 3s ...
This is my startup script:
gitlab-runner register
--non-interactive
--url <my gitlab instance>
--registration-token <my token>
--tag-list runner-autoscale-test
--env "DOCKER_AUTH_CONFIG={\"credsStore\":\"ecr-login\"}"
--env "AWS_SDK_LOAD_CONFIG=true"
--executor docker+machine
--docker-tlsverify
--docker-image alpine:latest
--tls-cert-file /etc/ssl/private/localhost.bundle
--tls-key-file /etc/ssl/private/localhost.key
--docker-disable-cache
--docker-privileged
--cache-type "s3"
--cache-shared
--cache-path "/cache"
--cache-s3-server-address "s3.amazonaws.com"
--cache-s3-bucket-name <my bucket>
--cache-s3-bucket-location "ap-southeast-2"
--cache-s3-insecure=false
--machine-idle-nodes 1
--machine-idle-time 1800
--machine-max-builds 100
--machine-machine-driver amazonec2
--machine-machine-name gitlab-docker-machine-%s
--machine-machine-options amazonec2-region=ap-southeast-2
--machine-machine-options amazonec2-use-private-address
--machine-machine-options amazonec2-instance-type=m4.xlarge
--machine-machine-options amazonec2-tags=Name,gitlab-runner-instance,environment,preview
--machine-machine-options amazonec2-security-group=gitlab-runner-sg-preview
--machine-machine-options amazonec2-vpc-id=<my vpc>
--machine-machine-options amazonec2-request-spot-instance=false
--machine-machine-options amazonec2-iam-instance-profile=gitlab-autoscaling-runner-preview
--machine-machine-options amazonec2-ssh-user=ubuntu
--machine-machine-options amazonec2-ami=<my ami>
--machine-machine-options amazonec2-userdata=/etc/gitlab-runner/userdata.sh
Steps to reproduce
Use the gitlab-runner register
command above, where the image from .gitlab-ci.yml
pulls from ECR.
.gitlab-ci.yml
image: 1234567890.dkr.ecr.ap-southeast-2.amazonaws.com/my-ci:latest
stages:
- build
- release
variables:
DOCKER_AUTH_CONFIG: '{\"credsStore\":\"ecr-login\"}'
AWS_SDK_LOAD_CONFIG: 'true'
before_script:
etc
Actual behavior
Pulling docker image 1234567890.dkr.ecr.ap-southeast-2.amazonaws.com/my-ci:latest ...
ERROR: Preparation failed: Error response from daemon: Get https://1234567890.dkr.ecr.ap-southeast-2.amazonaws.com/v2/my-ci/manifests/latest: no basic auth credentials (executor_docker.go:175:0s)
Environment description
- Ubuntu 18.04.2 LTS (GNU/Linux 4.15.0-1043-aws x86_64)
- Docker version 18.09.7, build 2d0083d
config.toml contents
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = <name>
url = <url>
tls-cert-file = "/etc/ssl/private/localhost.bundle"
tls-key-file = "/etc/ssl/private/localhost.key"
executor = "docker+machine"
environment = ["DOCKER_AUTH_CONFIG={\"credsStore\":\"ecr-login\"}", "AWS_SDK_LOAD_CONFIG=true"]
[runners.custom_build_dir]
[runners.docker]
tls_verify = true
image = "alpine:latest"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = true
volumes = ["/cache"]
shm_size = 0
[runners.cache]
Type = "s3"
Path = "/cache"
Shared = true
[runners.cache.s3]
ServerAddress = "s3.amazonaws.com"
BucketName = <bucket>
BucketLocation = "ap-southeast-2"
[runners.cache.gcs]
[runners.machine]
IdleCount = 1
IdleTime = 1800
MaxBuilds = 100
MachineDriver = "amazonec2"
MachineName = "gitlab-docker-machine-%s"
MachineOptions = ["amazonec2-region=ap-southeast-2", "amazonec2-use-private-address", "amazonec2-instance-type=m4.xlarge", "amazonec2-tags=Name,gitlab-runner-instance,environment,preview", "amazonec2-security-group=gitlab-runner-sg-preview", "amazonec2-vpc-id=<vpc>", "amazonec2-request-spot-instance=false", "amazonec2-iam-instance-profile=gitlab-autoscaling-runner-preview", "amazonec2-ssh-user=ubuntu", "amazonec2-ami=<ami>", "amazonec2-userdata=/etc/gitlab-runner/userdata.sh"]
OffPeakTimezone = ""
OffPeakIdleCount = 0
OffPeakIdleTime = 0
Used GitLab Runner version
- gitlab-runner --version
Version: 12.0.1
Git revision: 0e5417a3
Git branch:
GO version: go1.8.7
Built: 2019-06-26T15:02:39+0000
OS/Arch: linux/amd64