[CI] Define a global variable for including files from a branch

Introduction

First of all, I hope that, despite the long description, the answer will be "You are doing it wrong. Here is how you should do it".

However, since I don't know better, I'll explain here with as many details I can provide.

Testing changes in the CI when using included files, especially on more than one nesting level, is by far the most complicated and time-consuming process.

It requires providing branch names almost everywhere, committing these changes, and removing them once I complete my changes.

Scenario

Below are a couple of YAML files taken from an actual project (just slightly sanitized).

my-group/my-front-end-project/.gitlab-ci.yml:

---
include:
  - project: 'my-group/my-cd-project'
    file:
      - '/gitlab-ci/front-end.yml'

my-group/my-cd-project/gitlab-ci/front-end.yml:

---
image: some-image

workflow:
  rules:
    - if: '$CI_COMMIT_MESSAGE =~ /-wip$/ || $CI_PIPELINE_SOURCE == "merge_request_event"'
      when: never
    - when: always

stages:
  - development
  - prepare
  - quality
  - tests
  - qa
  - staging
  - production
  - documentation

include:
  - project: 'my-group/my-cd-project'
    file:
      - '/gitlab-ci/global.yml'
      - '/gitlab-ci/before-script.yml'
      - '/gitlab-ci/after-script.yml'
      - '/gitlab-ci/prepare-cache.yml'
      - '/gitlab-ci/qa/duplication.yml'
      - '/gitlab-ci/integrations/build-succeeded.yml'

pages:
  extends: .cache-pull
  stage: documentation
  script:
    - rm -rf public
    - npm run doc:generate -- --output public
    - echo "Pages URL $CI_PAGES_URL"
  artifacts:
    paths:
      - public
  rules:
    - if: '$CI_COMMIT_REF_SLUG == "master"'

I want to make some changes in the CI (on several of the included files) and test them without pushing them to the production branch.

What I usually do is the following ¹:

1. Create a new branch called my-new-feature in my-group/my-front-end-project
2. Create a new branch called my-new-feature in my-group/my-cd-project
3. In my-group/my-front-end-project/.gitlab-ci.yml, add a ref: my-new-feature to the include keyword
4. Do the same in my-group/my-cd-project/gitlab-ci/front-end.yml
5. Do the same in all the files included in my-group/my-cd-project/gitlab-ci/front-end.yml if they also include files.
6. Repeat as long as there are nested included files (and as long as I need to modify them).
7. Cry if the included files spread across other projects (this is the last step because, when it happens, I usually realize it after dealing with all the previous steps).

Proposal

Define "root" variables that tell the runner to include files from a branch with a given name but fall back to another one, should that branch not exist.

E.g.:

---
variables:
  INCLUDE_FROM_REF: $CI_COMMIT_REF_SLUG
  INCLUDE_FROM_REF_FALLBACK: $CI_DEFAULT_BRANCH

On every include, the runner will first try to include the file from INCLUDE_FROM_REF and:

1. Try to include the file from INCLUDE_FROM_REF.
2. Fail with an error if there is no branch with this name and no provided INCLUDE_FROM_REF_FALLBACK.
3. If the previous step didn't fail, try to include the file from INCLUDE_FROM_REF_FALLBACK.
4. Fail with an error if there is no INCLUDE_FROM_REF_FALLBACK branch.

I imagine that the CI Linter could handle the failure, as the required branches are meant to exist before starting any pipeline.

1. Frustration often leads me to skip all of this and push my changes to the main branch, which I hate to do. ↩