Temp directory for @pages.tmp is not cleaned up

Summary

In some cases, temp files are not being cleaned up in the /var/opt/gitlab/gitlab-rails/shared/pages/@pages.tmp directory. We suspect this may be related to https://gitlab.com/gitlab-org/gitlab/-/issues/327581.

We recently discovered this behaviour on gitlab.com. Several years worth of residual files were using up a lot of inodes in the filesystem. See: gitlab-com/gl-infra/production#4197 (closed).

Steps to reproduce

We don't have a clear reproducer at this point. We suspect large pages projects to be a factor, but this has not been confirmed.

Example Project

Not applicable.

What is the current bug behavior?

Temporary directories may not be deleted.

What is the expected correct behavior?

Temporary directories are deleted eventually.

Relevant logs and/or screenshots

Evidence we collected in production is summarised here: gitlab-com/gl-infra/production#4197 (comment 550818253).

Output of checks

This bug happens on GitLab.com

Possible fixes

One possible approach here would be to have a periodic job that scans @pages.tmp (e.g. once per day), and deletes directories created before a cutoff (e.g. 7 days ago).

The code that is creating these temporary directories is: UpdatePagesService.