Shared runners running out of disk space
As seen in https://gitlab.com/gitlab-org/gitlab-ee/-/jobs/293422760, our shared runners have been running out of disk space from time to time.
@tmaczukin said:
As for the full disk - yesterday I've deployed a configuration change that trigger cleanups less frequently. Because recently it was too aggressive and was causing removal of containers in the middle of the job (the mysterious container abcd1234 can't be find error). I suspect that the less frequent cleanup is now causing more often disk filling. We're using a 25GB disk and re-using machines for GitLab CE/EE tests up to 30 times.
It seems that bumping the size of the disk could be a solution here.
I think they are configured at 25 GB now. Should we double that?