Skip to content

dial tcp: lookup docker on x.x.x.x:53: no such host error runner inside docker on armhf

Summary

When running gitlab-runner inside docker on an armhf platform the following error is thrown when trying to run docker login:

error during connect: Post http://docker:2375/v1.39/auth: dial tcp: lookup docker on x.x.x.x:53: no such host

It is worth noting that I tried the same setup on a amd64 host with none of the same issues.

Steps to reproduce

  • Use the following gitlab-runner docker container built from the following Dockerfile:
  • https://gitlab.com/chevpot/gitlab-ci-runner/blob/master/Dockerfile @ sha 962bdf636e63f52c89cada3e4688fe21ff24f2f9
  • Run on a armhf device, in my case with docker-compose:
version: '3.4'

services:
        gitlab-ci-runner:
                image: gitlab-ci-runner:latest
                container_name: gitlab-ci-runner
                env_file:
                        - .env
                volumes:
                        - gitdata:/etc/gitlab-runner
                        - /var/run/docker.sock:/var/run/docker.sock
                ports:
                        - 8093:8093
                restart: unless-stopped
volumes:
        gitdata:
                name: gitdata
                driver: local
                driver_opts:
                        type: none
                        o: bind
                        device: "${GITLAB_BASE}/config

Register runner against given project Run a ci pipeline using that runner

.gitlab-ci.yml 
# Use the docker image as everything is run within docker containers
image: docker:18.09

services:
  - docker:18.09-dind

variables:
  API_IMAGE_PATH: registry.gitlab.com/chevpot/gitlab-ci-runner
  DOCKERHUB_IMAGE_PATH: chevpot/gitlab-ci-runner
  GITLAB_CI_RUNNER: $API_IMAGE_PATH/gitlab-ci-runner
  # DOCKER_TLS_CERTDIR: "/certs"
  # DOCKER_HOST: tcp://docker:2378
  # DOCKER_DRIVER: overlay

.docker-build-gitlab-ci-runner-arm: &docker-build-gitlab-ci-runner-arm |
  docker build \
      --cache-from $GITLAB_CI_RUNNER:latest-armhf \
      -f Dockerfile \
      -t $GITLAB_CI_RUNNER:$CI_COMMIT_SHA-armhf \
      -t $GITLAB_CI_RUNNER:latest-armhf \
      -t $DOCKERHUB_IMAGE_PATH:latest-armhf \
      .

.docker-build-gitlab-ci-runner: &docker-build-gitlab-ci-runner |
  docker build \
      --cache-from $GITLAB_CI_RUNNER:latest \
      -f Dockerfile \
      -t $GITLAB_CI_RUNNER:$CI_COMMIT_SHA \
      -t $GITLAB_CI_RUNNER:latest \
      -t $DOCKERHUB_IMAGE_PATH:latest \
      .

stages:
  - build
  - version

build-gitlab-arm:
  stage: build
  tags:
    - arm
  script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - docker pull $GITLAB_CI_RUNNER:latest-armhf || true
    - *docker-build-gitlab-ci-runner-arm
    - docker push $GITLAB_CI_RUNNER:$CI_COMMIT_SHA-armhf
    - docker push $GITLAB_CI_RUNNER:latest-armhf
    - docker login -u $DOCKERHUB_USERNAME -p $DOCKERHUB_PASSWORD
    - docker push $DOCKERHUB_IMAGE_PATH:latest-armhf

version-arm:
  stage: version
  tags:
    - arm
  script:
    - docker pull $DOCKERHUB_IMAGE_PATH:latest-armhf
    - GITLAB_CI_VERSION=`docker run -p 8093:8093 -v $PWD/config:/etc/gitlab-runner -v /var/run/docker.sock:/var/run/docker.sock $DOCKERHUB_IMAGE_PATH:latest-armhf --version | grep "^Version:" | cut -d ":" -f2 | tr -d '\040\011\012\015'`
    - docker tag $DOCKERHUB_IMAGE_PATH:latest-armhf $DOCKERHUB_IMAGE_PATH:$GITLAB_CI_VERSION-armhf
    - docker login -u $DOCKERHUB_USERNAME -p $DOCKERHUB_PASSWORD
    - docker push $DOCKERHUB_IMAGE_PATH:$GITLAB_CI_VERSION-armhf

build-gitlab-x86:
  stage: build
  tags:
    - docker
  script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - docker pull $GITLAB_CI_RUNNER:latest || true
    - *docker-build-gitlab-ci-runner
    - docker push $GITLAB_CI_RUNNER:$CI_COMMIT_SHA
    - docker push $GITLAB_CI_RUNNER:latest
    - docker login -u $DOCKERHUB_USERNAME -p $DOCKERHUB_PASSWORD
    - docker push $DOCKERHUB_IMAGE_PATH:latest

version-x86:
  stage: version
  tags:
    - docker
  script:
    - docker pull $DOCKERHUB_IMAGE_PATH:latest
    - GITLAB_CI_VERSION=`docker run -p 8093:8093 -v $PWD/config:/etc/gitlab-runner -v /var/run/docker.sock:/var/run/docker.sock $DOCKERHUB_IMAGE_PATH:latest --version | grep "^Version:" | cut -d ":" -f2 | tr -d '\040\011\012\015'`
    - docker tag $DOCKERHUB_IMAGE_PATH:latest $DOCKERHUB_IMAGE_PATH:$GITLAB_CI_VERSION
    - docker login -u $DOCKERHUB_USERNAME -p $DOCKERHUB_PASSWORD
    - docker push $DOCKERHUB_IMAGE_PATH:$GITLAB_CI_VERSION

Actual behavior

The following error occurs when docker login is run: error during connect: Post http://docker:2375/v1.39/auth: dial tcp: lookup docker on x.x.x.x:53: no such host

Expected behavior

No error should be thrown

Relevant logs and/or screenshots

job log 
Running with gitlab-runner 12.3.0 (a8a019e0)
  on pi-runner fDQ-o9mB
Using Docker executor with image docker:18.09 ...
Starting service docker:18.09-dind ...
Pulling docker image docker:18.09-dind ...
Using docker image sha256:2442d824e7b5ebe83b2fc19bc8ad1da320f70a21b10b3d647dc019ea82b5f5f7 for docker:18.09-dind ...
Waiting for services to be up and running...

*** WARNING: Service runner-fDQ-o9mB-project-14130149-concurrent-0-docker-0 probably didn't start properly.

Health check error:
ContainerStart: Error response from daemon: Cannot link to a non running container: /runner-fDQ-o9mB-project-14130149-concurrent-0-docker-0 AS /runner-fDQ-o9mB-project-14130149-concurrent-0-docker-0-wait-for-service/service (executor_docker.go:1253:0s)

Service container logs:


*********

Pulling docker image docker:18.09 ...
Using docker image sha256:33a72f73198ad7c4d449f19bcbf6f474627cdec624426c234da9818aec75a438 for docker:18.09 ...
Running on runner-fDQ-o9mB-project-14130149-concurrent-0 via 07c031f191e9...
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/chevpot/gitlab-ci-runner/.git/
Created fresh repository.
From https://gitlab.com/chevpot/gitlab-ci-runner
 * [new ref]         refs/pipelines/89323614 -> refs/pipelines/89323614
 * [new branch]      master                  -> origin/master
Checking out 962bdf63 as master...

Skipping Git submodules setup
$ docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
error during connect: Post http://docker:2375/v1.39/auth: dial tcp: lookup docker on 192.168.0.254:53: no such host
ERROR: Job failed: exit code 1

Environment description

The runner is being run within a custom docker container on an armhf host raspi v4, raspbian buster

config.toml contents 
concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "test-vm"
  url = "https://gitlab.com/"
  token = "FGvopuFEZCZBwHxTsa5V"
  executor = "docker"
  environment = ["DOCKER_TLS_CERTDIR="]
  [runners.custom_build_dir]
  [runners.docker]
    tls_verify = false
    image = "docker:stable-dind"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
  [runners.cache]
    [runners.cache.s3]

Used GitLab Runner version

Version:      12.3.0
Git revision: a8a019e0
Git branch:   12-3-stable
GO version:   go1.8.7
Built:        2019-09-20T08:09:48+0000
OS/Arch:      linux/arm

Possible fixes

Edited by Paul Topley