multiple cache in a same job

UPDATE

The MR for this issue has been merged under a feature flag: multiple_cache_per_job. You can follow the status of the rollout here: #321877 (closed)

Release notes

GitLab CI/CD provides a caching mechanism that can be used to save time when your jobs are running, previously it was impossible to configure multiple cache keys in the same job, this limitation caused users to abuse artifacts to be used as cache or duplicate the same job with different cache path, in this release we provide the ability to configure multiple cache keys in a single job which will help you increase your pipeline performance.

Problem to solve

Currently in CI pipelines we can use multiple cache using different keys. But we cannot use multiple cache on the same jobs.

In node.js projects we have two kinds of dependencies. The dev dependencies and prod dependencies. We need first to download (and cache) all dependencies. Later (before making a docker image) we need to remove dev dependencies.

Some additional reasons:

1. when multiple, parallel build stages use separate output directories that need to be cached, but the generated artifacts are all desired (or required) to go through a common test phase.
2. the use case in the bullet above would be more common in monorepos which are increasing in popularity.
3. when this is not implemented, GitLab artifacts end up being abused for this purpose since they will automatically aggregate all outputs of all jobs that do not have overlapping file or directory names for subsequent jobs.
4. when this is not implemented, specific job phases may need to be duplicated (e.g. test jobs) just because of the "single cache path per job" - which increases complexity and resource utilization.

Currently:

stages:

stages:
  - setup
  - build
  - test
  - pre_package
  - package
  - release

dev_dependencies:
  stage: setup
  cache:
    key: DEV-DEPENDENCIES-$CI_COMMIT_REF_SLUG
    paths:
      - node_modules/
    policy: pull-push
  script:
    # will download all dependencies
    - npm ci

prod_dependencies:
  stage: pre_package
  only:
    - master
  cache:
    key: PROD-DEPENDENCIES-$CI_COMMIT_REF_SLUG
    paths:
      - node_modules/
    policy: pull-push
  script:
    # will download prod dependencies
    - npm ci --only=production

Intended users

• Devon (DevOps Engineer)

Further details

Without this feature we need to download twice the dependencies rather to it once and benefit of cleaning feature of npm ci --only=production.

Proposal

prod_dependencies:
  stage: pre_package
  only:
    - master
  cache:
    - key: DEV-DEPENDENCIES-$CI_COMMIT_REF_SLUG
      paths:
        - node_modules/
      policy: pull
    - key: PROD-DEPENDENCIES-$CI_COMMIT_REF_SLUG
      paths:
        - node_modules/
      policy: push
  script:
    # will remove dev dependencies
    - npm ci --only=production

It would be cost saving to be able to use 2 cache rules in the same job.

Permissions and Security

non applicable

Documentation

?

Testing

If using two cache on same directory with same policy, what cache rule will be used first ?

What does success look like, and how can we measure that?

feature exists and works like expected

What is the type of buyer?

i don't know

Links / references

• CircleCI approach: https://circleci.com/docs/2.0/caching/#example-caching-configuration