Temp directory for @pages.tmp is not cleaned up
Summary
In some cases, temp files are not being cleaned up in the /var/opt/gitlab/gitlab-rails/shared/pages/@pages.tmp
directory. We suspect this may be related to https://gitlab.com/gitlab-org/gitlab/-/issues/327581.
We recently discovered this behaviour on gitlab.com. Several years worth of residual files were using up a lot of inodes in the filesystem. See: gitlab-com/gl-infra/production#4197 (closed).
Steps to reproduce
We don't have a clear reproducer at this point. We suspect large pages projects to be a factor, but this has not been confirmed.
Example Project
Not applicable.
What is the current bug behavior?
Temporary directories may not be deleted.
What is the expected correct behavior?
Temporary directories are deleted eventually.
Relevant logs and/or screenshots
Evidence we collected in production is summarised here: gitlab-com/gl-infra/production#4197 (comment 550818253).
Output of checks
This bug happens on GitLab.com
Possible fixes
One possible approach here would be to have a periodic job that scans @pages.tmp
(e.g. once per day), and deletes directories created before a cutoff (e.g. 7 days ago).
The code that is creating these temporary directories is: UpdatePagesService
.