Skip to content
GitLab
Next
Closed
Issue created by Steve Azzopardi@steveazzOwner

Automatically enable GCP DocerkHub mirror for DinD builds for shared runners

Problem

With the new Docker RateLimits users might start reaching the rate limits of pulling docker images. As discussed in https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11113 we aren't going to be affected by this because we are using the GCP mirror. However this mirror is only configured when pulling images for the job, it is not configured for the docker daemons that start docker in docker.

If users are using docker in docker to build their image, it is going to pull the base images it needs to build the image. Since the docker daemon that is started by dind is not configured to use the mirror it might reach some rate limits.

Proposal

Follow https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#docker-executor-inside-gitlab-runner-configuration by having our Shared runner fleet automatically mount /etc/docker/daemon.json that is configured to use GCP DockerHub mirror so even our docker in docker builds will use the mirror.

What we would need to do:

  1. Update the base VM image that is used in CI to include a daemon.json somewhere in the runner manager to have the config below.

    daemon.json 
    {
  "registry-mirrors": [
    "https://registry-mirror.example.com"
  ]
}

  2. Update the shared runner fleet configuration like below.

    config.toml 
    [[runners]]
  ...
  executor = "docker"
  [runners.docker]
    image = "alpine:3.12"
    privileged = true
    volumes = ["/opt/docker/daemon.json:/etc/docker/daemon.json:ro"]

Possible problems

When users have are configuring the mirror through command

This will be a breaking change for certain users

When a user has the following command defined in their .gitlab-ci.yml to specify a mirror (they can do this already) and we mount the /etc/docker/daemon.json the service is going to fail with the error below.

User updated docker dind service to specify a mirror themselves

services:
   - name: docker:19.03.13-dind
     command: ["--registry-mirror", "https://registry-mirror.example.com"] # Specify the registry mirror to use.

GitLab CI failure

2020-11-02T08:18:33.103369077Z unable to configure the Docker daemon with file /etc/docker/daemon.json: the following directives are specified both as a flag and in the configuration file: registry-mirrors: (from flag: [https://mirror.gcr.io/], from file: [https://registry-mirror.example.com])

Users don't expect the /etc/docker/daemon.json to be present

There might be some jobs out that that don't expect the /etc/docker/daemon.json to be present which might also break their jobs if we start mounting this file.

Edited by Steve Azzopardi