Excessive docker volume disk usage - docker executor

I just experienced a disk-full situation on a VM running gitlab-ci-multi-runner (v1.3.0 I think - I can confirm tomorrow). 28 GB of disk space was spent in /var/lib/docker/volumes.

From the Docker User Guide:

If you remove containers that mount volumes, ... the volumes will not be deleted. To delete the volume from disk, you must explicitly call docker rm -v against the last container with a reference to the volume. This allows you to upgrade, or effectively migrate data volumes between containers.

Note: Docker will not warn you when removing a container without providing the -v option to delete its volumes. If you remove containers without using the -v option, you may end up with “dangling” volumes; volumes that are no longer referenced by a container. You can use docker volume ls -f dangling=true to find dangling volumes, and use docker volume rm <volume name> to remove a volume that’s no longer needed.

Automatically cleanup orphaned Docker volumes

I was able to run this command to clean up all the orphaned volumes:

# docker volume ls -qf dangling=true | xargs -r docker volume rm

Perhaps the gitlab-ci-multi-runner daemon could periodically do just this?

Prevent the situation

The better option would be to not cause orphaned volumes in the first place.

It looks like the code in master does remove volumes when removing containers. I'm not sure if this code is not working, or if my volumes were left over from an older version which did not specify this option.

TO Do:

• Review issue, level of effort and priority for near term release milestone planning. Due Jan 27 2020 @DarrenEastman