## Summary We have a cache that is populated only by certain jobs. This is one of several caches we use; the others are relevant to all our jobs. Because there seems to be no good way to make this type of configuration on a per-job basis, we have all the caches configured for all jobs. I was assuming that when the target paths do not exist, the cache would not be created. However, while a warning is issued, it appears that Gitlab proceeds to create the (empty) cache anyway. I don't know why anyone would want an empty cache, but it gets worse. One would expect, even in this scenario, that the cache would eventually get filled because every job is configured `pull-push`. Once you run a relevant job on that runner you'd be successful. Even a job that doesn't touch the cache at all should leave it alone, and thus be a no-op with respect to the cache state. However, there appears to be a race when runners are configured to allow multiple jobs. You can end up in this scenario: * Job A starts and reads the empty cache (this job is not configured to build the cache) * Job B starts and reads the empty cache (this job is configured to build the cache) * Job B fills the cache and writes it * Job A completes and writes the empty cache, throwing away the previous value Thus it is possible to be in a perpetual state where the cache is not filled, even in steady state. All of this would be avoided if the cache is simply not written when it is empty. Clearly the code exists to detect this state (because a warning is issued), so make this warning actionable and choose not to write the cache when empty. Aside: even if you *eventually* converge on a filled cache, implementing my suggestion above would improve the speed of convergence in filling the cache. ## Demonstration Here's an example of some jobs that show the problem. First, we have a job that creates an empty cache, despite the fact that the target directories do not exist at all: https://gitlab.com/StanfordLegion/legion/-/jobs/10308350860#L3368 Here is a job where the runner "successfully" finds the cache: https://gitlab.com/StanfordLegion/legion/-/jobs/10308350860#L3368 But it is empty and contains no files: https://gitlab.com/StanfordLegion/legion/-/jobs/10308351006#L397 To demonstrate that there is a race and *sometimes* the cache gets filled, here's one where we do see its contents (and later on in the build, we successfully reuse the cache contents): https://gitlab.com/StanfordLegion/legion/-/jobs/10308350840#L52 ## Steps to reproduce In principle you need the following to trigger this: * A `.gitlab-ci.yml` with multiple jobs * One or a subset of jobs write the cache * The cache is configured globally * The runner is configured to allow multiple jobs to run concurrently You can see our `.gitlab-ci.yml` for inspiration: https://gitlab.com/StanfordLegion/legion/-/blob/3c3e9a78f1c682e9821967c6417e74a5a7b5f69d/.gitlab-ci.yml#L34-40 ## Actual behavior Runners race on creating the cache, resulting in either full or empty caches depending on who wins. ## Expected behavior Runners that do not find files to cache should not attempt to write it, allowing runners with full caches to win the race. (Those runners will still potentially race between themselves, but that is fine.) ## Environment description Project lives at https://gitlab.com/StanfordLegion/legion Runners are custom Docker runners configured for local cache. We do not have a shared cache enabled. Presumably, that would be even worse given the race that exists in the system. ### Used GitLab Runner version ```console $ gitlab-runner --version Version: 18.0.2 Git revision: 4d7093e1 Git branch: 18-0-stable GO version: go1.23.6 X:cacheprog Built: 2025-05-21T22:00:57Z OS/Arch: linux/amd64 ``` ## Possible fixes I suggest not writing the cache when no files are found. I'm not sure why writing an empty cache is useful anyway.