Docker executor is unable to connect to a local MinIO container
Summary
gitlab-runner with the Docker executor doesn't appear to be able to use a local MinIO container as a cache without hacks.
Steps to reproduce
I'm trying to put together a Docker-compose environment to run a runner with a shared cache on a MinIO container.
docker-compose.yml
(Each component is taken or adapted from official documentation for the specific software)
version: '3.7'
services:
gitlab-runner-cache:
container_name: gitlab-runner-cache
image: minio/minio
volumes:
- ${CACHE_DATA}:/data
environment:
MINIO_ACCESS_KEY: ${MINIO_ACCESS_KEY}
MINIO_SECRET_KEY: ${MINIO_SECRET_KEY}
command: server /data
gitlab-runner:
container_name: gitlab-runner
image: gitlab/gitlab-runner
volumes:
- ${RUNNER_CONF}:/etc/gitlab-runner
- /var/run/docker.sock:/var/run/docker.sock
config.toml
: (private bits omitted)
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "Local runner"
url = "https://gitlab.com/"
token = "XXX"
executor = "docker"
[runners.docker]
tls_verify = false
image = "ubuntu:16.04"
privileged = false
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
[runners.cache]
Type = "s3"
Shared = true
[runners.cache.s3]
ServerAddress = "gitlab-runner-cache:9000"
AccessKey = "XXX"
SecretKey = "XXX"
BucketName = "runner-cache"
BucketLocation = "us-east-1"
Insecure = true
Minimal .gitlab-ci.yml
cache:
paths:
- cache/
cache_test:
stage: test
before_script:
- mkdir -p cache
script:
- echo "This should be cached" > cache/cached_file.txt
Actual behavior
This configuration fails to download or upload data from the cache server as it cannot resolve gitlab-runner-cache
within the specific container used to pull the cache data from the server.
Expected behavior
I'd expect that as the MinIO container is on the same network as the container that extracts the cache, it'd be able to resolve gitlab-runner-cache
and retrieve cache data from it.
Relevant logs and/or screenshots
job log excerpt showing failure to retrieve cache
(Edited to redact unfiltered private data - the access and secret keys aren't filtered out in the error messages.)
Checking cache for branches-production...
WARNING: Retrying... error=Get http://gitlab-runner-cache:9000/runner-cache/project/[FILTERED]/branches-production?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=[FILTERED]&X-Amz-Date=20190819T045928Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=[FILTERED]: dial tcp: lookup gitlab-runner-cache on 192.168.65.1:53: no such host
WARNING: Retrying... error=Get http://gitlab-runner-cache:9000/runner-cache/project/[FILTERED]/branches-production?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=[FILTERED]&X-Amz-Date=20190819T045928Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=[FILTERED]: dial tcp: lookup gitlab-runner-cache on 192.168.65.1:53: no such host
FATAL: Get http://gitlab-runner-cache:9000/runner-cache/project/[FILTERED]/branches-production?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=[FILTERED]&X-Amz-Date=20190819T045928Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=[FILTERED] dial tcp: lookup gitlab-runner-cache on 192.168.65.1:53: no such host
Failed to extract cache
Used GitLab Runner version
Running with gitlab-runner 12.1.0 (de7731dd)
Possible fixes
I've managed to work around this by:
- Adding both containers to the default "bridge" network (Adding
network_mode: "bridge"
to both containers) - Exporting port 9000 of the MinIO container
- Using
host.docker.internal
instead ofgitlab-runner-cache
However this doesn't scale as it only works on Docker For Windows and Docker For MacOS as they provide that particular name within containers whereas "real" Docker on Linux doesn't.