Job execution hangs when latest docker:dind (version 19.03.6 and "floating" ones) is used
Summary
Gitlab pipeline timeout after given timeout seconds and saying ERROR: Job failed: execution took longer than 1h0m0s seconds
because it is failing to pull image from gitlab/gitlab-runner-helper
. Can you help as we are stuck here to deploy things live.
Workaround
Update your usage of docker:dind
image to a specific version docker:19.03.5-dind
for example:
image: docker:19.03.1
variables:
# When using dind service, we need to instruct docker, to talk with
# the daemon started inside of the service. The daemon is available
# with a network connection instead of the default
# /var/run/docker.sock socket. docker:19.03.1 does this automatically
# by setting the DOCKER_HOST in
# https://github.com/docker-library/docker/blob/d45051476babc297257df490d22cbd806f1b11e4/19.03.1/docker-entrypoint.sh#L23-L29
#
# The 'docker' hostname is the alias of the service container as described at
# https://docs.gitlab.com/ee/ci/docker/using_docker_images.html#accessing-the-services.
#
# Note that if you're using GitLab Runner 12.7 or earlier with the Kubernetes executor and Kubernetes 1.6 or earlier,
# the variable must be set to tcp://localhost:2376 because of how the
# Kubernetes executor connects services to the job container
# DOCKER_HOST: tcp://localhost:2376
#
# Specify to Docker where to create the certificates, Docker will
# create them automatically on boot, and will create
# `/certs/client` that will be shared between the service and job
# container, thanks to volume mount from config.toml
DOCKER_TLS_CERTDIR: "/certs"
services:
- docker:19.03.5-dind # <------ We use a specific version of docker in docker
before_script:
- docker info
build:
stage: build
script:
- docker build -t my-docker-image .
- docker run my-docker-image /script/to/run/tests
Root cause
- GitHub issues:
As explained in https://github.com/moby/moby/issues/40514#issuecomment-586282433:
Our linux shared Runners use a single core machine to run the job, which explains why users on GitLab.com using our Runners expreienced this.
Preliminary testing looks to confirm that it's related to number of cores; single-core machines hit this issue, and 2 or more cores don't
Edited by Steve Xuereb