Gitlab Runner/Docker not able to resolve DNS after job failed

Summary

We've noticed that sometimes after jobs fails, gitlab-runner/docker, when executing a new job, is no longer able to resolve our server hostname. Although entry is present in /etc/hosts file, resolving works from localhost/cmd and "dns" entries exist in both gitlab-runner config file + docker - it just stops working. In order to fix it, we simply need to run "systemctl restart docker" and then it works.

Steps to reproduce

Job basically failed because of the requirements - e.g. we were including PHP version 7.2 (or greater) in composer and that was not installed, 7.0 was available. So basically any error on purpose.

.gitlab-ci.yml

# Select image from https://hub.docker.com/_/php/
image: php:7.0

# Describe pipeline stages (not required)
stages:
  - build
  - test
  - deploy

# Runs before script in each stage 
before_script:
  # Install Git, the PHP image doesn't have it installed
  - apt-get update -qq
  - apt-get install -qq git

  # Setup SSH deploy keys
  - 'which ssh-agent || ( apt-get install -qq openssh-client )'
  - eval $(ssh-agent -s)
  - ssh-add <(echo "$SSH_PRIVATE_KEY")
  - mkdir -p ~/.ssh
  - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'

  # Install Composer
  - curl -sS https://getcomposer.org/installer | php

  # Install all project dependencies
  - php composer.phar install

.build:
  stage: build
  script:
    - echo "Building..."
    - php --version

test:
  stage: test
  script:
    - echo "Testing..."

deploy:
  stage: deploy
  environment:
    name: staging
    url: ci-cd-test.DOMAIN.TLD
  script:
    - echo "Deploying..."
    - ssh c1_ci_cd_ssh@SERVER_IP "cd /web/ && git checkout master && git pull origin master && exit"
  only:
    - master
#  tags:
#    - deploy
  when: on_success

Actual behavior

Gitlab-runner job fails with message: fatal: unable to access 'http://gitlab-ci-token:[MASKED]@git01.DOMAIN.TLD:8190/ci-cd-test-group/ci-cd-test.git/': Could not resolve host: git01.DOMAIN.TLD

Expected behavior

Resolving hostname should be possible without need to restart docker service.

Relevant logs and/or screenshots

job log

Running with gitlab-runner 12.4.1 (05161b14)
  on git01-runner zxzdxPuq
Using Docker executor with image php:7.0 ...
Pulling docker image php:7.0 ...
Using docker image sha256:cd45c80ef616aad81f96fd3fb46e795f5088fba365a0316eb7dcf6e55a36ced0 for php:7.0 ...
Running on runner-zxzdxPuq-project-3-concurrent-0 via test2.DOMAIN.TLD...
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/ci-cd-test-group/ci-cd-test/.git/
fatal: unable to access 'http://gitlab-ci-token:[MASKED]@git01.DOMAIN.TLD:8190/ci-cd-test-group/ci-cd-test.git/': Could not resolve host: git01.DOMAIN.TLD
ERROR: Job failed: exit code 1

Environment description

config.toml contents

concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "git01-runner"
  url = "http://git01.DOMAIN.TLD:8190/"
  token = "zxzdxPuqmdciqp4uQVQV"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.docker]
    tls_verify = false
    image = "alpine:latest"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
    dns = ["8.8.8.8", "1.1.1.1"]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]


ALSO
cat /etc/docker/daemon.json 
{
    "dns": ["8.8.8.8", "1.1.1.1"]
}

Used GitLab Runner version

gitlab-runner --version

Version: 12.4.1

Git revision: 05161b14

Git branch: 12-4-stable

GO version: go1.10.8

Built: 2019-10-28T12:49:57+0000

OS/Arch: linux/amd64

Possible fixes

When it's not working, when I do this:

docker run -it alpine ping git01.DOMAIN.TLD

ping: bad address 'git01.DOMAIN.TLD'

But when I run:

systemctl restart docker

afterwards it works:

docker run -it alpine ping git01.DOMAIN.TLD

PING git01.DOMAIN.TLD (SERVER_IP): 56 data bytes

64 bytes from SERVER_IP: seq=0 ttl=64 time=0.091 ms

64 bytes from SERVER_IP: seq=1 ttl=64 time=0.119 ms

64 bytes from SERVER_IP: seq=2 ttl=64 time=0.106 ms

^C

And that being said - if we restart above job - it succeed! We no longer get error "Could not resolve host: git01.DOMAIN.TLD".

Any idea how to debug this?