Force-removing Docker containers is leaving orphaned files behind, which can/will fill up a file system
Summary
https://github.com/docker/docker/issues/22207 describes this bug best. Basically, there's a Docker bug that's existed at least since last year that causes files to be left behind on the file system after force-removing a Docker image. This issue manifested itself on my GitLab server by completely filling up the root file system. I had to manually go in and delete the orphaned files in /var/lib/docker/aufs
.
According to the discussion:
me I'm not manually using docker at all. We're using gitlab-ci-multi-runner. Could it be a bug on GitLab's end then?
thaJeztah It looks like (by default) it force-removes containers; https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/blob/dbdbce2848530df299836768c8ea01e209a2fe40/executors/docker/executor_docker.go#L878. Doing so can result in failures to remove the container being ignored, and leading to the orphaned diffs.
me Ok, then that tells me that this is a gitlab-ci-multi-runner bug. Is that a correct interpretation? I'm happy to create an issue for them to fix this.
thaJeztah It's a combination I guess; "force" remove makes it easier to handle cleanups (i.e., cases where a container isn't stopped yet, etc), at the same time (that's the "bug" @cpuguy83 mentioned), it can also hide actual issues, such as docker failing to remove the containers filesystem (which can have various reasons). With "force", the container is removed in such cases. Without, the container is left around (but marked "dead")
If the gitlab runner can function correctly without the force remove, that'll probably be good to change (or make it configurable)
cgebe I am using Drone and have the same issue. I didn't check the code how containers are removed, but i guess it force removes as well.
Steps to reproduce
I think you can just enable Docker-based CI on a relatively small file system and then just use CI repeatedly over a period of time, and you should see orphaned files created in /var/lib/docker/aufs
.
Actual behavior
The file system fills up with orphaned files.
Expected behavior
The file system shouldn't fill up with orphaned files.
Relevant logs and/or screenshots
See the issue on Docker's GitHub (linked above).
Environment description
Are you using shared Runners on GitLab.com? Or is it a custom installation? Which executors are used? Please also provide the versions of related tools like
docker info
if you are using the Docker executor.
Latest Ubuntu LTS, latest Docker, using a shared runner on our custom GitLab install.
$ sudo docker info
Containers: 12
Running: 0
Paused: 0
Stopped: 12
Images: 7
Server Version: 17.03.1-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 72
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-70-generic
Operating System: Ubuntu 16.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.793 GiB
Name: git
ID: WIW7:7BIY:6NBU:E6DV:KFAY:4OAL:QTL5:VWSR:FXKU:FKS6:WN5O:AHEV
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Http Proxy: http://xxx.xxx.xxx:8080/
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Used GitLab Runner version
$ sudo gitlab-runner --version
Version: 9.0.0
Git revision: 08a9e6f
Git branch: 9-0-stable
GO version: go1.7.5
Built: Wed, 22 Mar 2017 16:29:52 +0000
OS/Arch: linux/amd64