multiple cache in a same job
UPDATE
The MR for this issue has been merged under a feature flag: multiple_cache_per_job
. You can follow the status of the rollout here: #321877 (closed)
Release notes
GitLab CI/CD provides a caching mechanism that can be used to save time when your jobs are running, previously it was impossible to configure multiple cache keys in the same job, this limitation caused users to abuse artifacts to be used as cache or duplicate the same job with different cache path, in this release we provide the ability to configure multiple cache keys in a single job which will help you increase your pipeline performance.
Problem to solve
Currently in CI pipelines we can use multiple cache using different keys. But we cannot use multiple cache on the same jobs.
In node.js projects we have two kinds of dependencies. The dev dependencies and prod dependencies. We need first to download (and cache) all dependencies. Later (before making a docker image) we need to remove dev dependencies.
Some additional reasons:
- when multiple, parallel build stages use separate output directories that need to be cached, but the generated artifacts are all desired (or required) to go through a common test phase.
- the use case in the bullet above would be more common in monorepos which are increasing in popularity.
- when this is not implemented, GitLab artifacts end up being abused for this purpose since they will automatically aggregate all outputs of all jobs that do not have overlapping file or directory names for subsequent jobs.
- when this is not implemented, specific job phases may need to be duplicated (e.g. test jobs) just because of the "single cache path per job" - which increases complexity and resource utilization.
Currently:
stages:
stages:
- setup
- build
- test
- pre_package
- package
- release
dev_dependencies:
stage: setup
cache:
key: DEV-DEPENDENCIES-$CI_COMMIT_REF_SLUG
paths:
- node_modules/
policy: pull-push
script:
# will download all dependencies
- npm ci
prod_dependencies:
stage: pre_package
only:
- master
cache:
key: PROD-DEPENDENCIES-$CI_COMMIT_REF_SLUG
paths:
- node_modules/
policy: pull-push
script:
# will download prod dependencies
- npm ci --only=production
Intended users
Further details
Without this feature we need to download twice the dependencies rather to it once and benefit of cleaning feature of npm ci --only=production
.
Proposal
prod_dependencies:
stage: pre_package
only:
- master
cache:
- key: DEV-DEPENDENCIES-$CI_COMMIT_REF_SLUG
paths:
- node_modules/
policy: pull
- key: PROD-DEPENDENCIES-$CI_COMMIT_REF_SLUG
paths:
- node_modules/
policy: push
script:
# will remove dev dependencies
- npm ci --only=production
It would be cost saving to be able to use 2 cache rules in the same job.
Permissions and Security
non applicable
Documentation
?
Testing
If using two cache on same directory with same policy, what cache rule will be used first ?
What does success look like, and how can we measure that?
feature exists and works like expected
What is the type of buyer?
i don't know
Links / references
- CircleCI approach: https://circleci.com/docs/2.0/caching/#example-caching-configuration
This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.