CI rules parameter hard to generalize across jobs

Problem to solve

GitLab 12.3 introduced the rules parameter, which can be used in lieu of only/except. I'm trying to use this in order to take advantage of rules:exists (introduced in GitLab 12.4). But, I'm finding the syntax impossible to generalize across multiple jobs with extends.

Here is a simplified example configuration before rules. Here, I have only/when conditions defined in a skipped job named .dev_conditions, which is extended by 3 concrete jobs named Plan, Apply, and Destroy, varying in when they're run.

.dev_conditions:
  only:
    refs:
      - branches
    variables:
      - $DEPLOYMENT_ENVIRONMENTS =~ /\bdev\b/
  except:
    refs:
      - master

Plan:
  extends:
    - .tf_plan
    - .dev_conditions
  when: always

Apply:
  extends:
    - .tf_apply
    - .dev_conditions
  when: on_success

Destroy:
  extends:
    - .tf_destroy
    - .dev_conditions
  when: manual

If I modify .dev_conditions to use the rules parameter instead of only/except, there's no easy way for me to vary the when parameter. I'd have to copy the conditions to each job.

.dev_conditions:
  rules:
    - if: …
      when: always
    - when: never

Plan:
  extends:
    - .tf_plan
    - .dev_conditions
  
Apply:
  extends:
    - .tf_apply
    - .dev_conditions
  # How to set when: on-success without copying the entire rules list?

Destroy:
  extends:
    - .tf_destroy
    - .dev_conditions
  # How to set when: manual without copying the entire rules list?

Intended users

• Devon (DevOps Engineer)

Further details

Jobs like Apply and Destroy extend .dev_conditions. But, because rules is a list, I can't override just the when parameter of any particular rule. I have to redefine the entire rules list.

Note: Since keywords like branches aren't supported in rules, this particular example isn't implementable using rules, regardless of whether when can be overridden.

Proposal

I'd like both the basic when parameter and the when attribute in rules to accept variables instead of just literal values. Then, a variable could be used for the when attribute in .dev_conditions, with the value provided by the jobs extending it.

Permissions and Security

N/A

Documentation

N/A

Testing

N/A

What does success look like, and how can we measure that?

Be able to convert from existing only/except parameters to rules and easily change the when attribute of rules without repeating entire rules lists.

What is the type of buyer?

Rules is part of Core.

Links / references

mentioned in issue #35127 (closed)

added devopsverify grouppipeline execution typefeature labels

Setting Category:Continuous Integration based on ~"group::continuous integration".

added Category:Continuous Integration label

mentioned in issue #35493 (closed)

In general, the YAML anchoring and templating doesn't support merging of "lists." Since the new rules syntax is structued as a list in YAML, it's difficult to template on this keyword in the CI pipeline.

We use a monorepo setup and have to copy/paste each only:changes config into a new anchored yaml format. Since in a monorepo set up you occassionally want to build all services for changes in certain root files, we have just copied and pasted the files in each template, for each service. It's very messy unfortunately.

I see the same issue here. If you are templating your pipelines, you have to put the rules keyword in either the template or the actual definition. You can't put them in both, since one overwrites the other.

Unless someone else knows of another way, this currently isn't possible. There is a similar issue for the only/changes syntax here: #8177 (closed)

The best fix is to either:

a) allow dynamic variable interpolation across all fields in the rules syntax OR

b) allow YAML lists to be merge-able in some way (this would be tricky with ordering and detecting duplicates).

mentioned in issue #35987 (closed)

mentioned in issue #34272 (closed)

added to epic &2783

mentioned in issue #209711 (closed)

changed milestone to %Backlog

added [deprecated] Accepting merge requests label

Not being able to merge rules would break all of our pipelines as we compose our jobs from a collection of included templates. We need to be able to filter a job out based on ref type and if certain variables are set or have a value, only/when worked well for this.

We're a Premium customer.

marked this issue as related to #218444

added CI rules label

We also find rules inferior to only/except for this.

It's easy to extend only/except to keep a common set of rules, but add something like changes

.docker-defaults:
  image: <private dind image>
  services:
    - docker:19.03.0-dind
  variables:
    IMAGE_NAME: $CI_REGISTRY_IMAGE/$project_name
    DOCKER_HOST: tcp://docker:2375
    DOCKER_DRIVER: overlay2
  before_script:
    - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN registry.gitlab.com
    
.build-docker-image:
  extends: .docker-defaults
  stage: build
  only:
    refs:
      - production
      - staging
  variables:
    IMAGE_TAG: $CI_REGISTRY_IMAGE/$project_name/scan:$CI_COMMIT_SHA
  script:
    - echo "build logic here"

.node:
  only:
    changes:
      - dockerfiles/cipipeline/alpine/0_node/**/*
      - dockerfiles/cipipeline/alpine/build_promote_docker.gitlab-ci.yml

build:node:
  extends: 
    - .build-docker-image
    - .node
  stage: build

.docker-defaults and .build-docker-image can exist in separate ci files, reusable across multiple projects.

Perhaps rules would be better as key/value pair definitions like:

rules:
  any:
    branch_production: 
      - $CI_COMMIT_BRANCH == "production"
    branch_staging: 
      - $CI_COMMIT_BRANCH == "staging"
  all:
    change_to_dockerfile: 
      changes: 
        - relative/path/to/Dockerfile

If done this way each rule has a Name that can be included in output for ease of debugging. To maintain some backward compatibility you could allow rules: to be a list or a key/value pair, or perhaps just add explicit rules_list and rules_named.

Stumbling on this issue while searching for a solution... I'll keep waiting fort it.

Sadly, I cannot advise any of my clients to migrate to rules: with this limitation. only/except are working great thought, I don't find myself limited.

Keep up the good work on this wonderful product

added sectionops label

added grouppipeline authoring + 1 deleted label and removed grouppipeline execution label

changed epic to &4399

Current Gold customer interested in this feature - https://gitlab.my.salesforce.com/0016100001ef6rH

added customer label

mentioned in merge request !34346 (closed)

Facing similar issues as the above commenters.

Also facing these issues. There are many other people getting confused by this:

• Directly relates to: #229980 (closed)
• Seems to be related: #219437 (closed), #213050, #322992 (closed), #35093, #16376 (closed), gitlab-foss#32973 (closed), gitlab-foss#38297 (closed), #254229 (closed), #266173 (closed)
• https://forum.gitlab.com/t/extends-with-rules/41292
• https://stackoverflow.com/questions/64423281/multiple-pipelines-for-the-same-branch
• https://stackoverflow.com/questions/67009007/how-to-merge-rules-on-a-gitlab-ci-job

In the example below, job-level rules totally overwrites rules from a hidden job referenced by extends.

.no-scheduled:
  rules:
    - if: '$CI_PIPELINE_SOURCE == "schedule"'
      when: never

apply:
  stage: apply
  needs: ["plan"]
  extends: .no-scheduled
  rules:
    - if: '$CI_COMMIT_REF_PROTECTED == "true"'
  script:
    - gitlab-terraform apply

Merged YAML result from Pipeline Editor

---
".no-scheduled":
  rules:
  - if: $CI_PIPELINE_SOURCE == "schedule"
    when: never
apply:
  rules:
  - if: $CI_COMMIT_REF_PROTECTED == "true"
  stage: apply
  needs:
  - plan
  extends: ".no-scheduled"
  script:
  - gitlab-terraform apply

This seems to be an issue related to how YAML merges lists and arrays as pointed by @phillycheeze in #35093 (comment 242524424) don't know how to help but still posting this for context.

EDIT: Worth pointing, !reference didn't work either:

  rules:
    - !reference [.no-scheduled, rules]

Error by CI Editor: jobs:apply rules should be an array of hashes.
Other people with the same issue: #322992 (closed) and https://forum.gitlab.com/t/using-reference-in-rules/52350

I'd be interested as to why the rules sections were designed as arrays rather than maps? As @1mamute correctly states the overriding behaviour is caused by the yaml array merging logic.

I am testing with using !reference in rules.

https://docs.gitlab.com/ee/ci/yaml/README.html#reference-tags 3

source:
  rules:
    - if: '$GITLAB_CI == "true"'
  script:  
    - echo hi

test:
  rules:
    - !reference [source, rules]
  script:  
    - echo hi

With the above config I get the following error.

This GitLab CI configuration is invalid: jobs:test rules should be an array of hashes.

I’m not sure if this is a bug in the docs (it doesn’t say where !reference can/can’t be used) or a bug in the !reference for rules itself.

Rules are one of the places I find a lot of duplication, but always with some variation. This means the !reference option is of great value.

I’m not sure if this is a bug in the docs (it doesn’t say where !reference can/can’t be used) or a bug in the !reference for rules itself.

@jeffcook thanks for sharing. I think the core issue is well described with #322992 (comment 615060158)

Technically you have $CI_JOB_NAME available so your still in a "rules if clause" bit of fun. If 'master' is your default branch you could change that to $CI_DEFAULT_BRANCH.

rules:
  - if: $CI_COMMIT_BRANCH != 'master' && $CI_JOB_NAME == 'Plan'
    when: always
  - if: $CI_COMMIT_BRANCH != 'master' && $CI_JOB_NAME == 'Apply'
    when: on_success
  - if: $CI_COMMIT_BRANCH != 'master' && $CI_JOB_NAME == 'Destroy'
    when: manual
  - when: never

If you don't want to stack conditions you could do something like:

rules:
  # Don't run on master
  - if: $CI_COMMIT_BRANCH == 'master || $CI_PIPELINE_SOURCE == 'schedule' # trigger etc ...
    when: never
  # Do run on Plan: always
  - if: $CI_JOB_NAME == 'Plan'
    when: always
  # Do run on Apply: on_success
  - if: $CI_JOB_NAME == 'Apply'
    when: on_success
  # Do run on Destory: manual
  - if: $CI_JOB_NAME == 'Destroy'
    when: manual
  # Never ever, ever on anything else
  - when: never

Other option would be to play with the needs: clause but with rules unless you want 'blocked' as a pipeline status on 'Destroy' you'd also have to add allow_failure: true.

You could use the needs: to chain dependencies if your working inner stage or use stages to break up what you want to happen.

Apply:
  needs: ['Plan']

Destroy:
  needs: ['Apply']

Then your core rule could rule more of them without needing all the extensions. Just food for thought.

There's still a core problem with rules being a list of "OR" rules and the only method of doing "AND" to be shell syntax of "&&".

Even with the !reference tag being supported to combine with other lists of rules.

with only/except it is very easy to add "AND" for only: changes with exiting refs: master and variables:

Additionally this behavior is the opposite of how Ansible handles this in yaml. Where the when: parameter can be a list but they are defaulted to "AND" with the "OR" logic having to be in line.

I prefer the current logic since it works more like everything else in existence (y tho Ansible).

Firewalls, router rules, etc... all tend to go with a thing -> conditional pattern so that logic can be self contained.

Adding chaining would be one way to solve something like this that is also commonly done in firewall (and other) rules.

added Category:Pipeline Composition label

We're also a GitLab customer facing this issue, please resolve this.

added sectionci label and removed sectionops label