docker+machine gitlab executor with private ECR repo: `no basic auth credentials`

Summary

I am trying to setup an environment where I have docker+machine spawning runners, where the images in .gitlab-ci.yml are stored in ECR.

After reading this - https://docs.gitlab.com/ce/ci/docker/using_docker_images.html#using-credential-helpers, I built an ami containing docker-credential-ecr-login in path /usr/bin/local

The problem I have now is ensuring that the dockerd on that machine is able to utilise this. Note that I am relying on the role assumed by the ec2 that is spawned to contain the authorisation to read from ECR.

I am able to ssh into the machine created by gitlab-runner, and run docker pull 1234567890.dkr.ecr.ap-southeast-2.amazonaws.com/my-ci-image when I configure the following things:

  • export AWS_SDK_LOAD_CONFIG=true - documented by AWS as being required if using an assumed role

If you are working with an assumed role please set the environment variable: AWS_SDK_LOAD_CONFIG=true also.

  • Set ~/.docker/config.json on the spawned instance to contain :
{
	"credsStore": "ecr-login"
}

So that validates that docker-credential-ecr-login is present and works.

However I am having no luck getting the spawned runner to pull the image when running the job. I have tried a few different ways to specify DOCKER_AUTH_CONFIG, including as a variable, and as a --env option during gitlab-runner register. I see:

Running with gitlab-runner 12.0.1 (0e5417a3)
  on <my gitlab bastion host> KKhxk4ma
Using Docker executor with image 1234567890.dkr.ecr.ap-southeast-2.amazonaws.com/my-ci:latest ...
WARNING: Container based cache volumes creation is disabled. Will not create volume for "/cache"
Pulling docker image 1234567890.dkr.ecr.ap-southeast-2.amazonaws.com/my-ci:latest ...
ERROR: Preparation failed: Error response from daemon: Get https://1234567890.dkr.ecr.ap-southeast-2.amazonaws.com/v2/my-ci/manifests/latest: no basic auth credentials (executor_docker.go:175:0s)
Will be retried in 3s ...

This is my startup script:

gitlab-runner register 
--non-interactive 
--url <my gitlab instance>
--registration-token <my token> 
--tag-list runner-autoscale-test 
--env "DOCKER_AUTH_CONFIG={\"credsStore\":\"ecr-login\"}" 
--env "AWS_SDK_LOAD_CONFIG=true" 
--executor docker+machine 
--docker-tlsverify 
--docker-image alpine:latest 
--tls-cert-file /etc/ssl/private/localhost.bundle 
--tls-key-file /etc/ssl/private/localhost.key 
--docker-disable-cache 
--docker-privileged 
--cache-type "s3" 
--cache-shared 
--cache-path "/cache" 
--cache-s3-server-address "s3.amazonaws.com" 
--cache-s3-bucket-name <my bucket> 
--cache-s3-bucket-location "ap-southeast-2" 
--cache-s3-insecure=false 
--machine-idle-nodes 1 
--machine-idle-time 1800 
--machine-max-builds 100 
--machine-machine-driver amazonec2 
--machine-machine-name gitlab-docker-machine-%s 
--machine-machine-options amazonec2-region=ap-southeast-2 
--machine-machine-options amazonec2-use-private-address 
--machine-machine-options amazonec2-instance-type=m4.xlarge 
--machine-machine-options amazonec2-tags=Name,gitlab-runner-instance,environment,preview 
--machine-machine-options amazonec2-security-group=gitlab-runner-sg-preview 
--machine-machine-options amazonec2-vpc-id=<my vpc>
--machine-machine-options amazonec2-request-spot-instance=false 
--machine-machine-options amazonec2-iam-instance-profile=gitlab-autoscaling-runner-preview
--machine-machine-options amazonec2-ssh-user=ubuntu 
--machine-machine-options amazonec2-ami=<my ami>
--machine-machine-options amazonec2-userdata=/etc/gitlab-runner/userdata.sh

Steps to reproduce

Use the gitlab-runner register command above, where the image from .gitlab-ci.yml pulls from ECR.

.gitlab-ci.yml 
image: 1234567890.dkr.ecr.ap-southeast-2.amazonaws.com/my-ci:latest

stages:
  - build
  - release

variables:
  DOCKER_AUTH_CONFIG: '{\"credsStore\":\"ecr-login\"}'
  AWS_SDK_LOAD_CONFIG: 'true'

before_script:
etc

Actual behavior

Pulling docker image 1234567890.dkr.ecr.ap-southeast-2.amazonaws.com/my-ci:latest ...
ERROR: Preparation failed: Error response from daemon: Get https://1234567890.dkr.ecr.ap-southeast-2.amazonaws.com/v2/my-ci/manifests/latest: no basic auth credentials (executor_docker.go:175:0s)

Environment description

  • Ubuntu 18.04.2 LTS (GNU/Linux 4.15.0-1043-aws x86_64)
  • Docker version 18.09.7, build 2d0083d
config.toml contents 
concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = <name>
  url = <url>
  tls-cert-file = "/etc/ssl/private/localhost.bundle"
  tls-key-file = "/etc/ssl/private/localhost.key"
  executor = "docker+machine"
  environment = ["DOCKER_AUTH_CONFIG={\"credsStore\":\"ecr-login\"}", "AWS_SDK_LOAD_CONFIG=true"]
  [runners.custom_build_dir]
  [runners.docker]
    tls_verify = true
    image = "alpine:latest"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = true
    volumes = ["/cache"]
    shm_size = 0
  [runners.cache]
    Type = "s3"
    Path = "/cache"
    Shared = true
    [runners.cache.s3]
      ServerAddress = "s3.amazonaws.com"
      BucketName = <bucket>
      BucketLocation = "ap-southeast-2"
    [runners.cache.gcs]
  [runners.machine]
    IdleCount = 1
    IdleTime = 1800
    MaxBuilds = 100
    MachineDriver = "amazonec2"
    MachineName = "gitlab-docker-machine-%s"
    MachineOptions = ["amazonec2-region=ap-southeast-2", "amazonec2-use-private-address", "amazonec2-instance-type=m4.xlarge", "amazonec2-tags=Name,gitlab-runner-instance,environment,preview", "amazonec2-security-group=gitlab-runner-sg-preview", "amazonec2-vpc-id=<vpc>", "amazonec2-request-spot-instance=false", "amazonec2-iam-instance-profile=gitlab-autoscaling-runner-preview", "amazonec2-ssh-user=ubuntu", "amazonec2-ami=<ami>", "amazonec2-userdata=/etc/gitlab-runner/userdata.sh"]
    OffPeakTimezone = ""
    OffPeakIdleCount = 0
    OffPeakIdleTime = 0

Used GitLab Runner version

  • gitlab-runner --version
Version:      12.0.1
Git revision: 0e5417a3
Git branch:
GO version:   go1.8.7
Built:        2019-06-26T15:02:39+0000
OS/Arch:      linux/amd64

Possible fixes

Edited by jwieren