CI pipeline with parallel jobs sharing the same cache can result in an incomplete cache

Summary

A pipeline which runs jobs in parallel which share the same cache can produce non deterministic results in the cache resp. some files may be missing in subsequent jobs.

It looks there is a race condition between the concurrent jobs

• job 1 starts prepares the cache (local file deletion and extraction from pulled cache)
• job 2 is finished and builds the cache archive at the same time job 1 deletes cache, so an incomplete archive is built and cached files get lost

To prevent this cache extraction and creation need to be synchronized. They must not run at the same time (at least on the same runner).

Steps to reproduce

Since this is a race condition it's not easy to reproduce this behavior reliably. But I have a .gitlab-ci.yml at hand and screenshots, which show an erroneous and expected result.

This example pipeline just initializes the cache by deleting all files in the first stage and add one file to it. Then two jobs start in parallel and add different subdirectories. The last stages just shows the content of the cache folder.

stages:
  - cache-init
  - cache-add
  - cache-show

cache-init:
  stage: cache-init
  cache:
    key: stuff
    paths:
      - cache
  script:
    - rm -rf cache
    - mkdir -p cache
    - date > cache/index.txt

cache-1:
  stage: cache-add
  cache:
    key: stuff
    paths:
      - cache
  script:
    - mkdir -p cache/1
    - date > cache/1/stuff.txt

cache-2:
  stage: cache-add
  cache:
    key: stuff
    paths:
      - cache
  script:
    - mkdir -p cache/2
    - date > cache/2/stuff.txt

cache-show:
  stage: cache-show
  cache:
    key: stuff
    policy: pull
    paths:
      - cache
  script:
    - ls -lR cache

What is the current bug behavior?

This screenshot shows the output of job cache-show. The folder 2 of the job cache-2 is missing.

What is the expected correct behavior?

This screenshot shows the output of job cache-show with a complete cache directory.

Results of GitLab environment info

I don't have shell access to our GitLab server, so I can't run the environment commands, sorry.

GitLab CE: 10.3.2.

Runner: gitlab-ci-multi-runner, 9.5.1, shell runner