CI Job Failed - no space left on device (executor_docker.go:745:0s)
$ eval "$CI_PRE_CLONE_SCRIPT"
00:00
Downloading archived master...
Connecting to storage.googleapis.com (74.125.31.128:443)
gitlab.tar.gz 19% |****** | 156M 0:00:04 ETA
wget: short write: No space left on device
ERROR: Job failed (system failure): Error response from daemon: symlink ../8d32976da964751868f5ded7eded8bfb1de485402acbfd9094a3662bee40b249-init/diff /var/lib/docker/overlay2/l/CIZT2Z7HZJ32ERCMYKV2DGFJE6: no space left on device (executor_docker.go:745:0s)
Job:
Runner:
- gitlab-shared-runners-manager-6.gitlab.com
As noted in the details bellow, we've found that the issue is caused by the amount of forks and changes that are targeting GitLab. Every MR for every fork generates CI jobs. For many forks there are also jobs triggered for updates mirrored automatically from the original fork. On the runner's autoscaled VM it ends with "almost the same" git data being cloned over and over again to different cached volumes, that are uniq for every fork project.
Details can be read at #197876 (comment 274982928).
We've decided to go with the second solution proposed at the linked comment - to utilize the GIT_CLONE_PATH
feature and bring all of the community forks working directories to one place, that will be shared between jobs (please remember, that most of GitLab project's jobs are executed on gitlab-shared-runners-manager-X
runners, which re-use the autoscaled VMs; this is the main source of our problem here
With that configuration, we should highly reduce disk space usage, since instead of 10/20/30 copies of GitLab repo (which cloned even with GIT_DEPTH: 20
, but with all of the leftovers like bundle install
result, may use up to 1GB of space) that is stored in a cache volume, we will have it stored once. And re-used for every fork's job that happens to land on the specific VM.
We will of course leave our main forks (and especially the security ones!) on the "normal", non-shared paths, so that any secrets that we define will be not leaked.
TODO
-
close !24887 (merged) with all of its requirements - this will implement usage of shared working directory for GitLab and GitLab FOSS forks, -
wait a week or two for the change to propagate across many of the existing and all new MRs, -
repeat checks from #197876 (comment 274982928) to see if the change brought us the benefit that we expect from it.