Skip to content

2023-12-15: docker:24.0.7-dind failing to start on GitLab Runner (saas)

Customer Impact

Jobs failing to build images, for example: ERROR: error during connect: Get "http://docker:2375/_ping": dial tcp: lookup docker on 169.254.169.254:53: no such host

We have identified the root cause which is effecting CI jobs using the docker image (https://hub.docker.com/_/docker) with the latest tag release 24.0.7 or tag latest.

Workaround is to downgrade your image tag to a previous revision e.g 24.0.6.
This can be done by updating your Gitlab CI configuration yaml.

# .gitlab-ci.yml
job:
  # downgrade image tag
  image: docker:24.0.6
  # if you are using services with docker-in-docker
  services:
    - docker:24.0.6-dind

Current Status

As of 20:45 UTC Docker image have been updated with the fix from https://github.com/docker-library/docker/pull/465. Functionality to use revert back to iptables if nf_tables is not available have been added. Customers are now able to use lastest and 24.0.7 tags for Docker

The latest docker 24.0.7 has updated the alpine image https://github.com/docker-library/docker/pull/461 which updates the iptables version to use nf_tables. Our infrastructure currently doesn't support nf_tables. We are working with the upstream community in https://github.com/docker-library/docker/issues/463 since this issue is widespread and there is a potential fix in https://github.com/docker-library/docker/pull/465

cos@runner-j2nyww-s-s-l-s-amd64-1702624868-cb0c326e ~ $ docker run --privileged --entrypoint=/sbin/iptables -it docker:24.0.6-dind --version
iptables v1.8.9 (legacy)
cos@runner-j2nyww-s-s-l-s-amd64-1702624868-cb0c326e ~ $ docker run --privileged --entrypoint=/sbin/iptables -it docker:24.0.7-dind --version
iptables v1.8.10 (nf_tables)

📚 References and helpful links

Recent Events (available internally only):

Deployment Guidance

Use the following links to create related issues to this incident if additional work needs to be completed after it is resolved:

Note: In some cases we need to redact information from public view. We only do this in a limited number of documented cases. This might include the summary, timeline or any other bits of information, laid out in our handbook page. Any of this confidential data will be in a linked issue, only visible internally. By default, all information we can share, will be public, in accordance to our transparency value.

Security Note: If anything abnormal is found during the course of your investigation, please do not hesitate to contact security.

Edited by Marcel Chacon