"docker run" always hangs when using docker in docker

Summary

"docker run" always hangs when using docker in docker

Steps to reproduce

The issue is reproduced automatically. See Example Project below.

1. In a GitLab.com CI job use a docker image based on alpine:edge.
2. Install docker client and daemon.
3. Run docker daemon in background.
4. Run a simple command in a new container: docker run busybox:latest echo hello.
5. Observe: the command hangs.

Example Project

See https://gitlab.com/abogdanenko/dockerhangs.

The project contains just two files so I'm including them below.

Dockerfile:

FROM alpine:edge
VOLUME /var/lib/docker
RUN apk upgrade && apk add --no-cache docker

.gitlab-ci.yml:

build-docker-image:
    image: docker
    script:
    - docker login --username "$CI_REGISTRY_USER" --password "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY"
    - docker build --tag "${CI_REGISTRY_IMAGE}/${CI_COMMIT_REF_NAME}" .
    - docker push "${CI_REGISTRY_IMAGE}/${CI_COMMIT_REF_NAME}"
    services:
    - docker:dind
    stage: build
docker-sanity-check:
    image: ${CI_REGISTRY_IMAGE}/${CI_COMMIT_REF_NAME}
    script:
    - dockerd &
    - sleep 10
    - docker run busybox:latest echo hello
    - killall dockerd
    - sleep 15
    - echo end-of-script
    stage: test
stages:
- build
- test

The job timeout for the project is set to 10 minutes.

What is the current bug behavior?

The following command prints "hello" but never completes:

docker run busybox:latest echo hello

What is the expected correct behavior?

The following command completes in a reasonable time (less than 2 minutes):

docker run busybox:latest echo hello

Relevant logs and/or screenshots

$ docker run busybox:latest echo hello
...
ERROR: Job failed: execution took longer than 10m0s seconds

Output of checks

This bug happens on GitLab.com

The demo project produces a docker image registry.gitlab.com/abogdanenko/dockerhangs/edge:latest. I pulled the image to my computer, created a container and ran docker client inside the container. The client didn't hang. So I guess the issue is somehow specific to GitLab.com.

Info about Docker versions:

1. alpine:edge provides Docker 19.03.
2. alpine:latest provides Docker 18.09 and does not have this problem.
3. Ubuntu 19.10 provides Docker 19.03 and does not have this problem either.

mentioned in issue #35516 (closed)

added Category:Continuous Integration devopsverify severity3 typebug labels

added 1 deleted label

removed 1 deleted label

Setting ~"group::continuous integration" based on Category:Continuous Integration.

added grouppipeline execution label

mentioned in issue #35987 (closed)

moved to gitlab-runner#4882 (closed)