Usage of variables in only: changes: paths

mentioned in issue #8189 (closed)

Wish this was a bug and was possible, but: Not actually bug as per: https://docs.gitlab.com/ee/ci/variables/where_variables_can_be_used.html

They are:

Supported for definitions where the "Expansion place" is:
Runner.
Script execution shell.

Not supported:
For definitions where the "Expansion place" is GitLab.
In the only and except variables expressions.

As i understand it, Variables aren't expanded until the job is actually run, so you can't use variables in only/except, except in the 'variables:' part (but only non-job variables i.e. project variables)

edit: Should be noted that I am not affiliated with Gitlab, just a fellow user who ran into the same problem, wishing it would work and then stumbling on the linked documentation

Pragmatically, you're obviously right, it doesn't work and in not working is "working as designed".

However I don't read the docs as being quite as definitive. The "not supported" section you cite refers to persisted variables. What do the first two entries in this table mean?

https://docs.gitlab.com/ee/ci/variables/where_variables_can_be_used.html#gitlab-ciyml-file

It says that the expansion place is in GitLab rather than in the Runner. Which does imply that some expansion happens before we get to the Runner.

mentioned in issue #8214 (closed)

mentioned in issue #8233 (closed)

mentioned in issue #8250 (closed)

mentioned in issue #8266 (closed)

mentioned in issue #8268 (closed)

mentioned in issue #8277 (closed)

mentioned in issue #8294 (closed)

mentioned in issue #8321 (closed)

mentioned in issue #8344 (closed)

mentioned in issue #8365 (closed)

mentioned in issue #8377 (closed)

mentioned in issue #8381 (closed)

mentioned in issue #8389 (closed)

mentioned in issue #8404 (closed)

mentioned in issue #8420 (closed)

mentioned in issue #8430 (closed)

added devopsverify pipeline + 1 deleted label

This would be very very handy indeed!

mentioned in issue #11765 (closed)

added Category:Continuous Integration label

Yep, very very handy, even more with "extends" key word

.deploy:
  stage: deploy
  only:
    changes:
      - template-${TIER}.yaml
  script:
    - deploy

my-front:
  extends: .deploy
  variables:
    TIER: front

my-back:
  extends: .deploy
  variables:
    TIER: back

If I change the template of my back, it won't redeploy my front by example. Very very useful feature with docker & k8s

+1

How do we make this a feature request?

Yes indeed, exactly my use case. How indeed do we make that feature request. ANy Gitlab folks out there to help us?

added typefeature label

added auto updated potential proposal labels

added [deprecated] Accepting merge requests label

added grouppipeline execution label

removed [deprecated] Accepting merge requests label

mentioned in issue #13384 (closed)

mentioned in issue #13456 (closed)

added Enterprise Edition label

mentioned in issue #13612 (closed)

changed milestone to %Backlog

added [deprecated] Accepting merge requests label

It would be very nice if we could use $CI_JOB_NAME in the only-changes part of a job. It would then be possible to have:

.build
  only:
    refs:
     - merge_request
    changes:
     - path/to/files/$CI_JOB_NAME/**/*
  script:
    - build path/to/files/$CI_JOB_NAME

prog1:
  extends: .build

prog2:
  extends: .build  

prog3:
  extends: .build

If prog1, prog2, and prog3 are built in an identical way, this way of writing makes for a short an easy to read .gitlab-ci.yml file.

Just tried for hours to get this to work, only to realize that it's not supported in the changes section.

+1

+1 hopes changes would have supported variables.

+1 its very useful if you have a large monorepo

+1

mentioned in issue #25855 (closed)

@jlenny Does this make sense to include as part of &1237 (closed)? It seems like a good way to make CI powerful using concepts that we already have. Maybe only to build into rules:if, since I think we want to focus on improving that syntax and deprecating only/except.

It's a nontrivial amount of work, because variables interpolation is completely built into the syntax parsing and evaluation of only:variables/rules:if and that would need to be extracted somehow to use in changes, but from a usage standpoint I think it makes sense to do.

Also would love to see this implemented. We use a monorepo, and having to hardcode the folder name for each service causes over a hundred lines in our gitlab-ci.yaml file to be duplicated.

Maybe this can be linked to this epic: &812 (closed)

mentioned in issue #35093

added to epic &1237 (closed)

+1

@thaoyeager this is one worth looking at scheduling - it has a lot of interest in a short time and is maybe relatively simple.

Hey @thaoyeager did you have a chance to take a look at this one?

@jyavorska As Drew mentioned in his comment, would we want to support this feature request regarding variables with rules:if rather than add to only/except?

Yes, I think that's the way we'd want to go @thaoyeager. One thing we should double check is make sure that our documentation for everything under only/except is helping people discover rules:if.

Thanks @jyavorska

+1

I'm facing the same limitation when creating a pipeline for a monorepo.

Is there any reliable workaround for achieve this?

I've switched to using jsonnet for describing gitlab-ci jobs, I found it much more usable in this case.

mentioned in issue #34272 (closed)

marked this issue as related to #34272 (closed)

A 200-seat premium customer is interested (Zendesk, internal use only) in this feature.

added customer label

This appears to be a special case of the broader issue gitlab-runner#1809 (closed), which is being investigated for a solution. Please join the discussion there, or please let me know if I've missed something and this actually is a separate issue.

marked this issue as a duplicate of gitlab-runner#1809 (closed)

closed

marked this issue as related to gitlab-runner#1809 (closed)

+1

+1! This feature would be hugely beneficial to monorepo setup.

mentioned in commit cinc-project/docker-images@f8ea2a5f

+1

+1 please, add for only:ref too

+1.. as @wengkhing mentioned it will be greatly useful in monolithic repos...

mentioned in issue gitlab-runner#1809 (closed)

+1

removed milestone %Backlog

+1

@jyavorska : Can you re-open this item? Obviously it is not a duplicate of gitlab-runner#1809 (closed) as no one in 1809 can explain why it would be a duplicate. Thanks :)

Edit: Adding @thaoyeager

Hey there, @thaoyeager is now PM for this area so it would be up to her.

+10

@thaoyeager Can you re-evaluate this? gitlab-runner#1809 (closed) has 344 and this has 64 votes respectively.

+1

This feature would be really neat with my use case for building and deploying OpenFaaS functions

.deploy-fn:
  stage: deploy
  only:
    changes:
      - $CI_JOB_NAME/package.json
      - $CI_JOB_NAME/pom.xml
  script:
    - ./faas-cli up --tag=sha --filter "$CI_JOB_NAME"

fn-do-all-the-things:
  extends: .deploy-fn

-> Every time someone updates the version of a function in the languages corresponding manifest file redeploy only the changed function to OpenFaaS

Can you please take a look at this issue @thaoyeager?

+1

reopened

changed due date to October 02, 2020

removed auto updated label

assigned to @thaoyeager

@DarrenEastman Can we consider this as resolved in %13.8 (cf. gitlab-runner#1809 (closed))?

assigned to @DarrenEastman

@tnir : #1809 does not solve this issue. But I think it might be a duplicate of #34272 (closed) (using the new "rules" syntax vs the old "only" syntax in this ticket). I also just found #214527 (closed) which might also be a duplicate of #34272 (closed).

removed [deprecated] Accepting merge requests label

added auto updated popular proposal labels

added sectionops label

mentioned in issue #254155 (closed)

mentioned in issue #257781 (closed)

mentioned in issue #261988 (closed)

mentioned in issue gitlab-org/quality/triage-reports#482 (closed)

mentioned in issue gitlab-org/quality/triage-reports#541 (closed)

mentioned in issue gitlab-org/quality/triage-reports#617 (closed)

mentioned in issue gitlab-org/quality/triage-reports#682 (closed)

mentioned in issue gitlab-org/quality/triage-reports#715 (closed)

mentioned in issue gitlab-org/quality/triage-reports#805 (closed)

mentioned in issue gitlab-org/quality/triage-reports#869 (closed)

changed milestone to %Backlog

added CI rules CI variables labels

+1

Having a large monorepo where all my services (and their path, dockerfile, etc) are referenced by variables, this would be really handy to manage my "only:changes:" conditions

+1

I would also LOVE to simplify my CI file by using a variable to reduce repeated code. My Use Case includes a folder for each "stack" (aka microservice) in the repo, and I am looking to use a CI file that contains the following:

.test_stack_template:
  tags:
    - micro
  stage: test
  script: "$STACK/test_service"
  only:
    changes:
      - $STACK/**/*

test_Example_Stack:
  extends: .test_stack_template
  variables:
    STACK: ExampleStackName

The desired result would be for the test_Example_Stack job to run when there are changes inside the folder with the name "ExampleStackName" or any of it's subfolders.

It is so weird we can use variables in cache or script but not in only. Is this targeted to be solved for some version? Or can this be archived with rules instead of only?

Variables can be used in rules: changes: - see #34272 (closed)

I wonder if my specific issue is related to using a variable in a hidden job that is extended by the job that sets the variable. In my usage, it works with the script, but not only:changes.

Have you tried this scenario before? (see my previous comment on this thread for an example)

unassigned @DarrenEastman

+1

This should be fixed once #293674 (closed) is addressed.

removed the relation with gitlab-runner#1809 (closed)

marked this issue as related to #293674 (closed)

+1

unassigned @thaoyeager

added [deprecated] Accepting merge requests label

added grouppipeline authoring label and removed grouppipeline execution label

added 1 deleted label

mentioned in issue #293674 (closed)

+1

+10

removed Category:Continuous Integration label

added candidate14.7 needs weight labels

wow it's now progressed, can't wait to see its presence

+1

added candidate14.9 label and removed candidate14.7 label

We would love to see this as well.

+1

removed due date

added candidate15.1 label and removed candidate14.9 label

Customer interested in this feature:

US Federal Ticket (Internal access to verified US Citizens only)
SFDC (internal link only)

+1

From 13.6, you can use the rules:changes syntax. Discussed in #19232 (closed).

mentioned in issue gitlab-org/ci-cd/pipeline-authoring#46 (closed)

@marknuzzo can we weight this issue?

Hi @furkanayhan - can you please review, refine as necessary and weigh this issue, please? Thank you!

This is an interesting problem...

Let's have this config;

".gitlab-ci.yml"

image: alpine

variables:
  CONFIG_PATH: "configs"

onlychanges:
  script: exit 0
  only:
    changes:
      - configs/*

onlychangeswithvar:
  script: exit 0
  only:
    changes:
      - $CONFIG_PATH/*

ruleschanges:
  script: exit 0
  rules:
    - changes:
      - configs/*

ruleschangeswithvar:
  script: exit 0
  rules:
    - changes:
      - $CONFIG_PATH/*

rulesexists:
  script: exit 0
  rules:
    - exists:
      - configs/*

rulesexistswithvar:
  script: exit 0
  rules:
    - exists:
      - $CONFIG_PATH/*

When you commit a change for a file configs/*.txt, this pipeline is created;

It means that; we don't expand variables for only:changes but we do expand variables for rules:changes. Yet, we don't expand variables for rules:exists. I guess we also need to solve this for rules:exits.

I wonder why we didn't implement variables for only:changes while doing it for rules:changes in #34272 (closed) & ! 45037. @mbobin Do you remember anything related to this?

cc @dhershkovitch @marknuzzo

#34272 (closed) was weighted as 3 so this can be the same probably.

@furkanayhan

I wonder why we didn't implement variables for only:changes while doing it for rules:changes

I noticed that our documentation states that only and except are not being actively developed. rules is the preferred keyword to control when to add jobs to pipelines.

Wonder if this is the reason why

@dhershkovitch Probably yes... So, maybe we should direct our users to rules:changes instead of fixing this

I can give the user perspective - there are still many examples in your docs using only:. This is probably why many users end up here.

And just to second this, there are docs about rules that refer back to the docs in only/except.

@dhershkovitch @marcel.amirault

I agree, earlier today I've opened this issue to try to fix it.

@furkanayhan I think each keyword has it own class responsible for the evaluation(from what I remember) and we need to implement one by one. Here is the issue for rules:exists: #283881 (closed)

changed the description

set weight to 3

removed needs weight label

mentioned in issue #357580 (closed)

removed candidate15.1 label

+1

In my case I'm able to use a variable inside rules:changes

use rules instead of only ;)

job_1:
  image: busybox
  stage: stage_1
  variables:
    SERVICE_NAME: service_1
    FOLDER: foo
  extends:
    - .rules
  script:
    - *****

.rules:
      - if: $CI_PIPELINE_SOURCE == "push"
        when: manual
        changes:
          - root_folder_a/${FOLDER}/${SERVICE_NAME}/**/*
          - root_folder_a/${FOLDER}/*.md
        allow_failure: true
      - if: $CI_PIPELINE_SOURCE == "web"
        when: manual
        allow_failure: true

but the variables you used aren't defined by the user, they are predefined variables from gitlab

In changes the variables are defined by me ;)

I'm also able to make this work, so perhaps this issue should be closed. I've got matrixed jobs building docker images like so:

.imgdirs: &image_directories
    - DIRECTORY: 'deploy/alpine'
    - DIRECTORY: 'deploy/node-14-alpine'
    - DIRECTORY: 'node/14-alpine'
    - DIRECTORY: 'node/16-alpine'
    - DIRECTORY: 'node/18-alpine'

build-images:
  parallel:
    matrix: *image_directories
  stage: build
  extends: .dind
  script:
    - ...build script...
  rules:
    - changes:
      - images/${DIRECTORY}/**/*

I'm pretty sure it didn't work when I had ./images/${DIRECTORY}/**/* (leading ./), which would seem strange, but perhaps I had something else misconfigured .

I confirm that with leading PATH (./) or variables, it doesn't work. Thank you for the heads up, prolly saved me hours of headache ^^

I'm trying to do something similar: I have a repository for all my Docker images and each image has it's own directory.

I've configured a matrix build job listing each image as a list item of a variable as each image should be build the same way and I want to avoid code duplication wherever possible.

I've configured my build job like the following:

build:
  stage: build
  parallel:
    matrix:
      - IMAGE_NAME:
          - image-1
          - image-2
          - image-3
  script: "./build-image.sh ${IMAGE_NAME}"
  only:
    changes:
      - .gitlab-ci.yml
      - build-image.sh
      - ${IMAGE_NAME}/**/*

That should be quite similar to that example from @jamesmontalvo3. It also works whenever I manually trigger the pipeline (or by schedule). But the pipeline is never triggered by changes in the sub-directory.

So, I guess I'm affected by the issue, that this does not work if the only.changes rule is starting with the variable name as mentioned by @jean-keumar, right? I.e. something like ${IMAGE_NAME}/**/* does not work but images/${IMAGE_NAME}/**/* should work?

Edit: Moving my image folders to a images sub-folder and using images/${IMAGE_NAME}/**/* in the only.changes rule also does not trigger the job whenever anything changes in the sub-folder of the image (i.e. changing the Dockerfile does not trigger the build job).

Edit 2: Got it working by specifying the IMAGE_NAME as in the example from @jamesmontalvo3 (i.e. repeating the variable name for each image):

build:
  stage: build
  parallel:
    matrix:
      - IMAGE_NAME: image-1
      - IMAGE_NAME: image-2
      - IMAGE_NAME: image-3
...

Now, whenever something in the image sub-folder changes, the job for the image is started as expected. I.e. changing the Dockerfile for image-2 triggers a job for image-2.

mentioned in issue #338312 (closed)

+1

added workflowready for development label

removed [deprecated] Accepting merge requests label

mentioned in issue #387002

added Support Interest label

removed 1 deleted label

added grouppipeline security label and removed grouppipeline authoring label

added grouppipeline authoring label and removed grouppipeline security label

added grouppipeline security label and removed Category:Pipeline Composition grouppipeline authoring labels

mentioned in issue gitlab-org/quality/triage-reports#12007 (closed)

added grouppipeline authoring label and removed grouppipeline security label

removed CI variables label

added Category:Pipeline Composition label

Seems still to be an issue in 'nested' rules.

#338312 (comment 1405628195)

added sectionci label and removed sectionops label

I'm wondering if there's any plan to get it on the roadmap.

mentioned in issue gitlab-com/Product#13611

added co-create label

@dhershkovitch Since we've documented for a while now that only is deprecated and no longer being developed, perhaps we should close this issue? https://docs.gitlab.com/ee/ci/yaml/#only--except

i'm not sure, but this issue is still not working for rules so if u close this issue, you should open one for rules keyword I guess

@nchal Have you tried using rules:changes? It should work. Could you provide an example that does not work?

yes, my bad, everything's fine, thx

FYI, inputs already offer a partial solution to this request that might fit most of your use cases. See https://docs.gitlab.com/ee/ci/yaml/inputs.html for details

I would appreciate if you could share use cases when inputs is not a suitable solution. While we know its limitations, we don't know if they are actually important in this case.

changed the description

The CI/CD variables are supported for rules:changes and we advice users to migrate to the new syntax because the only and except keywords are deprecated.

cc @nmezzopera @marcel.amirault @shekharpatnaik

closed

Usage of variables in only: changes: paths

Clarification

Summary

Steps to reproduce

Example Project

What is the current bug behavior?

What is the expected correct behavior?

Relevant logs and/or screenshots

Output of checks

Results of GitLab environment info

Results of GitLab application Check

Possible fixes

Designs

Child items ...

Activity

Usage of variables in only: changes: paths

Clarification

Summary

Steps to reproduce

Example Project

What is the current bug behavior?

What is the expected correct behavior?

Relevant logs and/or screenshots

Output of checks

Results of GitLab environment info

Results of GitLab application Check

Possible fixes

Relates to

Activity