Support merging rules arrays with !reference

Release notes

The YAML function called !reference which was introduced early this year allowed you to reuse the same configuration in multiple jobs, however merging rules was not supported. In this release, we've added support for merging rules array !reference, which lets you target the rules configuration you want to reuse as apart of your CI/CD pipeline, even if it's in another file.

Summary

Issue #266173 (closed) introduced the !reference notation to re-use job sections. This works well for the script-related tags and some other cases as noted in the documentation, but in the case of rules doesn't work (and maybe some other places), or at least doesn't work as expected per the documentation.

What is the current bug behavior?

When trying to !refence a rules array a syntax error is reported. For example:

.no_schedule:
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"
      when: never
    - when: on_success

job1:
  image: alpine:latest
  rules:
    - !reference [.no_schedule, rules]
  script:
      - echo "run except schedules and tags"

fails linting with the error This GitLab CI configuration is invalid: jobs:job1 rules should be an array of hashes.

It appears, assuming the merged YAML view is correct (a great feature BTW), that this is because with the current implementation !reference doesn't flatten arrays merged into an array element, it inserts the referenced array creating a nested array. For example, this script:

.my_script:
  script:
    - echo "foo"
    - echo "bar"
  after_script:
    - echo $CI_COMMIT_BRANCH

job4:
  image: alpine:latest
  before_script:
    - !reference [.my_script, after_script]
  script:
    - !reference [.my_script, script]
    - echo "after foo and bar"
    - !reference [.my_script, after_script]

results in the following merged job with various nested arrays in before_script and script

:job4:
  :image: alpine:latest
  :before_script:
  - - echo $CI_COMMIT_BRANCH
  :script:
  - - echo "foo"
    - echo "bar"
  - echo "after foo and bar"
  - - echo $CI_COMMIT_BRANCH

The various script properties can accept nested arrays of strings (per the linter), so this executes as expected, but rules does not as noted in the error above.

The only ways to to use !reference with rules appears to be either reference only a rule list with syntax like that shown below, which does not allow merging of rules (at least I couldn't find a syntax that worked)

.no_schedule:
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"
      when: never
    - when: on_success

job2:
  image: alpine:latest
  rules: !reference [.no_schedule, rules]
  script:
    - echo "run except schedules"

or create individual rule hashes (not arrays) and reference those hashes as array elements

.only_no_schedule:
  rules:
    if: $CI_PIPELINE_SOURCE == "schedule"
    when: never

job3:
  image: alpine:latest
  rules:
    - if: $CI_COMMIT_TAG
      when: never
    - !reference [.only_no_schedule, rules]
    - when: on_success
  script:
    - echo "run except schedules and tags"

In the latter case it appears any existing hash key that can be resolved by !reference can also be used, for example:

.rule_references:
  no_schedule:
    if: $CI_PIPELINE_SOURCE == "schedule"
    when: never
  no_tag:
    if: $CI_COMMIT_TAG
    when: never

job5:
  image: alpine:latest
  rules:
    - !reference [.rule_references, no_schedule]
    - !reference [.rule_references, no_tag]
    - when: on_success
  script:
    - echo "run except schedules and tags"

What is the expected correct behavior?

The capability should exist to merge rules arrays with !reference.