Update 2023-10-18:
There is a known issue with using variable expansion in rules:exists when using folders (addressed in #417251 (closed)).
The recommended workaround is to break the subfolders out of the original variable, as shown in the example below:
variables:DIR:"folder"FILE:"flag"FULL_PATH:".folder/flag"a:rules:-exists:-$FULL_PATH.# this doesn't work job a is not added to the pipelineimage:alpine:latestscript:-echo hib:rules:-exists:-$DIR/$FILE.# this works job b is added to the pipelineimage:alpine:latestscript:-echo hi
The longer your gitlab-ci.yml scripts are, the more difficult they are to maintain and scale. By adding support for environment variables in the rules: exists keyword, you can now use variables for paths or filenames without making your CI file overly verbose. Variables help you reduce your configuration file's overall length when running the same CI jobs to test changes in different file sets.
Problem to solve
I have some CI jobs which run the same steps on different files. GitLab does not currently evaluate variables in rules:exists.
My standard hack for creating "empty" folders in Git is to put a .ignore file in the folder, which then gets committed & checked out OK, but the directory appears "empty" to most globs, etc. - hth!
@mbobin Oh really? Even those defined under variables: in my .gitlab-ci file? I remember trying that a few weeks ago and it did not work. Is that a new thing? Is it documented anywhere?
@mbobin but it doesn't work when the variable comes from dotenv artifacts. Any ideas?
For exmple this job:
publish_prod:image:alpinestage:publish-prodscript:-echo "hi.. please help me"-echo "VAR_FROM_DOTENV=$VAR_FROM_DOTENV"rules:-if:'$VAR_FROM_DOTENV=="custom_value"'needs:-job:job_with_env_varsartifacts:true
In another job called job_with_env_vars I create the artifacts/reports/dotenv with the variable VAR_FROM_DOTENV. That works well because if I comment the rule, and let the job publish_prod run and I print the variable VAR_FROM_DOTENV it shows the value I put in the previous job.
I think the issue is when the runner is trying to determine if this job can be included, the variables from dotenv hasn't been included.. I gess.
Because if put the variable in the variables section directly, (not from dotenv) it works as well as you show us in your example
@dacardona, it doesn't work because rules are evaluated when a job is created(more exactly, when the pipeline is created), so the job_with_env_vars job didn't execute to generate the dotenv artifacts.
Thanks @mbobin!! good to know that.. I dont't know if your explanation is in the documentation, I couldn't see it. If no, It would be very helpful to have it due many people (including me) don't know that key point.
To add a use case to this request: If my branch name is "myBranchName, then I would like to be able to create a file named ".rules/myBranchName.env", and then have a rules section that looks like:
This does not seem like a big ask, as the CI_COMMIT_REF_NAME variable should be scope when the "rules: exists:" section is being evaluated.
Fyi, our intent here is for developers to be able to control pipeline behavior without directly editing the .gitlab-ci.yml , but instead allow them to create "control files" which can influence pipeline behavior, for example when iterating on a set of changes which only need to run a few pipeline jobs
Use case here: I'm writing a job template for my front-end team to check their code style using ESLint. If there isn't an ESLint config file in the project, I create a default one.
Another use case:
I'm providing project templates for my developers, which mostly work in python.
Sometimes they'll have a docker image in their project, sometimes they won't.
I'd like the docker jobs to only work if there's a Dockerfile in the project, but Dockerfiles can be anywhere and have different names.
With this feature implemented I could provide a default through an environment variable (DOCKERFILE=./Dockerfile) that can be then overridden in special cases.
This would allow them to create pipelines that can automatically discover whether a job should be created in the first place, because otherwise, it would make the pipeline fail which in reality it just shouldn't be run;
Current workaround requires each downstream projects to define additional when: never which could be entirely avoided if we can check it in the templates - and make the template maintainable, since it's currently technically possible for us to explicitly define the exist check for each job, but it won't be friendly to future maintenance, also rules:change support variable expansion and we're using it there;
Are there any plans to pick this up in the next releases? Would appreciate any update @dhershkovitch
We have a general and modular pipeline defined in a "CI/CD includes" project... it has some job families which should be created only if certain directories exist in the project source code.
In particular this apply to deployment/testing jobs per environment, where we have environment deployment specs in separate directories...
Then deploy-1/main.yml creates job families for the different environments (e.g. dev-deploy, dev-stop, dev-acceptance-test, dev-overlay, etc.), but not all our projects have the configurations for the environments supported by the includes library, so we want those jobs to not be created (otherwise they fail, we could check from the scripts if the overlay dir exists and do nothing but then the pipeline is misleading where it shows a deployment to an environment which does not exist).
Why interested: We are in the process of standardizing our CI setups and being able to parameterize things like rules:exists with variables is important to creating robust, reusable CI configurations.
Problem they are trying to solve: We want to be able to use CI variables in rules:exists in order to skip jobs in a parameterized way.
Current solution for this problem: Use an if condition in the script to exit the CI job early if the relevant files do not exist. Because it's the job script, variables are available as normal.
Impact to the customer of not having this: Wasted CI runner time and longer build times because the job has to be created and a runner has to start just for it to exit without doing anything. If we could use exists, the job would not be created at all.
@dhershkovitch@marknuzzo I've enabled this feature flag on the gitlab project today, and will enable it in my personal project to test it. If all goes well I might enable it to 25% of actors in .com EOD to see if I can enable it globally before the code cutoff, although timeline is tight.
An array of file paths. Paths are relative to the project directory ($CI_PROJECT_DIR) and can’t directly link outside it. File paths can use glob patterns and CI/CD variables.
It seems, that even in the next major version 16 this is not yet implemented, even though the docs include this since version 15.6 (iirc). I cant get my variables to work properly in rules yet. Am I missing something?
@jawys if you're interested in making a community contribution to help speed things up, please let me know - happy to get some engineering support for an implementation guide.
@jocelynjane as paying customers, we expect this among other issues (especially #332501, which results in wasted runner minutes) to be dealt with over time.