Docker DIND Causing Random Hangs
Description
The dind
service in Gitlab CI is causing random hangs, usually on a RUN echo ...
command in our CI jobs. This appears to be a recent issue with an upgrade of the Docker engine in the dind
image.
Examples of hung Autoware.Auto jobs:
- https://gitlab.com/simon-t4/AutowareAuto/-/jobs/455105697
- https://gitlab.com/simon-t4/AutowareAuto/-/jobs/455069107
- https://gitlab.com/autowarefoundation/autoware.auto/AutowareAuto/-/jobs/455306868
Examples of hung Autoware.ai jobs:
- https://gitlab.com/autowarefoundation/autoware.ai/docker/-/jobs/455022106
- https://gitlab.com/autowarefoundation/autoware.ai/docker/-/jobs/454632209
- https://gitlab.com/autowarefoundation/autoware.ai/docker/-/jobs/449923797
Tickets have been created to log similar behavior here and here. The temporary fix seems to be pinning the version of the dind
image used, as described in this StackOverflow question.
How to Reproduce
Happens randomly on CI jobs that use dind
to build Docker images.
Current Behavior
CI randomly hangs, causing delays in development.
Expected behavior
CI jobs should always complete, even if unsuccessful.