Add conditional "needs" blocks for better DAG pipelines
Proposal
With the introduction of needs
blocks in .gitlab-ci
files, working with DAG pipelines finally started making sense. However, at the current state, there still are cases in which working with stages is needed.
As an example, here is a scenario that I need for my projects:
# This job downloads and caches npm dependencies only when we have changes in package-lock.json
# The goal is to avoid redownloading everything everythime we run a build
Download NPM Dependencies:
image: $NODE_IMAGE
cache:
key:
files:
- package-lock.json
paths:
- node_modules/
policy: push
rules:
- changes:
- package-lock.json
script:
- npm ci
Build:
image: $NODE_IMAGE
cache:
key:
files:
- package-lock.json
paths:
- node_modules/
# Since this job doesn't change NPM dependencies, we can just pull the cache
policy: pull
needs:
- Download NPM Dependencies
script:
- ...
The idea behind this scenario is fairly simple: if we have changes in our package-lock.json
file we need to redownload (and cache again) all the dependencies, if not we can just skip the stage and get to the build. The build, however, needs the cache to be populated and up to date, hence why it has a needs
block.
This works fine when we have changes in the package-lock.json
file, as both jobs exist, but when we have no changes to package-lock.json
an invalid yaml file is produced (the dependencies job doesn't exist in that case, so we're pointing to a non-existing job in our needs
block).
An easy way around this would be to put them into two different stages and drop the needs
block, but I'd like to have this working on a stageless pipeline. Another idea could be to duplicate the build job and mix both needs
and rules
to produce two different build jobs based on the changes: package-lock.json
condition, but I'm not a fan of duplicating things.
So here's my proposal: allow needs
blocks to have the same syntax as rules
. This will change my build job into something like:
[...]
Build:
image: $NODE_IMAGE
cache:
key:
files:
- package-lock.json
paths:
- node_modules/
policy: pull
needs:
- job: Download NPM Dependencies
rules:
- changes:
- package-lock.json
script:
- ...
As a result, that needs
block would be evaluated only if the rule is true, meaning that with no changes to package-lock.json
the build job wouldn't depend on anything else and thus it will be able to run.