gitlab spams /tmp dir in docker container with gitlab-pages
Summary
I have a GitLab CE instance running on an ubuntu server using the official gitlab-ce docker image. The usual data volumes are mounted to a host directory srv/gitlab
, which resides on a separate disk.
volumes:
- '/srv/gitlab/config:/etc/gitlab'
- '/srv/gitlab/logs:/var/log/gitlab'
- '/srv/gitlab/data:/var/opt/gitlab'
This morning I found the root drive 100% used, with the major amount beeing used by the running gitlab-ce docker container (10.1 GB).
Digging into it I found the /tmp
dir inside the container is 9.3 GB in size, being entirely made up of gitlab-pages-<sometimestamp>
directories (996 apparently).
After clearing out the /tmp
dir it immediately starts filling up again, at approx. 1-2 gitlab-pages-<sometimestamp>
directories per second!
I currently have one single repository which uses pages to publish a Sandcastle built library documentation on tagged builds. Since this has been established only 2 tags were added to the project.
The container is stopped and restarted daily by an automated maintenance procedure. I'm using GitLab CE 10.7.0.
What is the current bug behavior?
GitLab fills up the docker containers' /tmp
directory with gitlab-pages-<sometimestamp>
directories until the host disk runs out of space.
What is the expected correct behavior?
GitLab must not repeately spam files into the /tmp
dir.
The /tmp
dir contents should not grow endlessly but instead be cleaned up on a regular basis, at least on a container restart.
Also /tmp
files should be removed when they are no longer needed.
Relevant logs and/or screenshots
Results of docker system df -v
du -hsx /* | sort -rh | head -n 40
on the docker container root
Results of Expand for output
root@gitlab:/# du -hsx /* | sort -rh | head -n 40 9.3G /tmp 1.4G /opt 130M /usr 24M /var 24M /lib 7.4M /bin 3.5M /sbin 2.2M /etc 32K /run 28K /assets 12K /root 4.0K /srv 4.0K /mnt 4.0K /media 4.0K /lib64 4.0K /home 4.0K /boot 4.0K /RELEASE 0 /sys 0 /proc 0 /dev
du -hsx /* | sort -rh | head -n 40
on containers' /tmp
dir
Results of Expand for output
root@gitlab:/# du -hsx /tmp/* | sort -rh | head -n 20 9.6M /tmp/gitlab-pages-1524465583158923969 9.6M /tmp/gitlab-pages-1524465582125403367 9.6M /tmp/gitlab-pages-1524454211109324575 9.6M /tmp/gitlab-pages-1524454209747360893 9.6M /tmp/gitlab-pages-1524454208716069855 9.6M /tmp/gitlab-pages-1524454207684050608 9.6M /tmp/gitlab-pages-1524454206651237250 9.6M /tmp/gitlab-pages-1524454205383355475 9.6M /tmp/gitlab-pages-1524454204350860122 9.6M /tmp/gitlab-pages-1524454203312405992 9.6M /tmp/gitlab-pages-1524454202279725437 9.6M /tmp/gitlab-pages-1524454201247414286 9.6M /tmp/gitlab-pages-1524454200216047447 9.6M /tmp/gitlab-pages-1524454199185104114 9.6M /tmp/gitlab-pages-1524454198153812276 9.6M /tmp/gitlab-pages-1524454196871811221 9.6M /tmp/gitlab-pages-1524454195836429231 9.6M /tmp/gitlab-pages-1524454194805022391 9.6M /tmp/gitlab-pages-1524454193771707728 9.6M /tmp/gitlab-pages-1524454192737619754
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Current User: git Using RVM: no Ruby Version: 2.3.6p384 Gem Version: 2.6.13 Bundler Version:1.13.7 Rake Version: 12.3.0 Redis Version: 3.2.11 Git Version: 2.14.3 Sidekiq Version:5.0.5 Go Version: unknownGitLab information Version: 10.7.0 Revision: 2011c9b Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql URL: https://gitlab.ad.haprotec.de HTTP Clone URL: https://gitlab.ad.haprotec.de/some-group/some-project.git SSH Clone URL: git@gitlab.ad.haprotec.de:some-group/some-project.git Using LDAP: yes Using Omniauth: no
GitLab Shell Version: 7.1.2 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab ...Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 3/3 ... yes 3/4 ... yes 6/8 ... yes 6/9 ... yes 6/10 ... yes 9/11 ... yes 10/12 ... yes 8/13 ... yes 8/16 ... yes 10/17 ... yes 6/18 ... yes 6/19 ... yes 8/21 ... yes 11/22 ... yes 8/23 ... yes 9/24 ... yes 9/25 ... yes 9/26 ... yes 11/27 ... yes 15/28 ... yes 11/29 ... yes 16/30 ... yes 8/31 ... yes 17/32 ... yes 6/34 ... yes 9/35 ... yes 9/36 ... yes 11/37 ... yes 16/38 ... yes 16/39 ... yes 8/40 ... yes 6/41 ... yes 15/42 ... yes 9/43 ... yes 9/44 ... yes 27/45 ... yes 11/46 ... yes 6/47 ... yes 9/48 ... yes 11/49 ... yes 9/51 ... yes 22/53 ... yes 27/55 ... yes 9/56 ... yes 9/57 ... yes 9/58 ... yes 6/59 ... yes 29/60 ... yes 30/62 ... yes 29/63 ... yes 31/64 ... yes 32/65 ... yes 16/68 ... yes 15/69 ... yes 67/70 ... yes 32/71 ... yes 8/72 ... yes 69/73 ... yes 9/74 ... yes 16/75 ... yes 11/76 ... yes 11/77 ... yes 9/78 ... yes 9/79 ... yes 72/81 ... yes 71/82 ... yes 9/84 ... yes 22/85 ... yes 73/86 ... yes 75/87 ... yes 74/88 ... yes 75/89 ... yes 75/90 ... yes 76/93 ... yes 11/95 ... yes 9/96 ... yes 73/97 ... yes 29/98 ... yes 82/99 ... yes 82/100 ... yes 9/101 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.5 ? ... yes (2.3.6) Git version >= 2.9.5 ? ... yes (2.14.3) Git user has default SSH configuration? ... yes Active users: ... 22
Checking GitLab ... Finished