Multiple CI/CD stage groups per project

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

Description

In my current project I need to be able to define two separated pipelines per GitLab project:

• A pipeline for building, testing and publishing (build a Docker image and publish it to our Docker registry) the app on every commit on master
• A pipeline for deploying the published app into our infrastructure(s)

The reason for this separation is that we've several apps (one GitLab project per app) and an upstream product GitLab project that contains a pipeline for setting up our infrastructure (Infrastructure as code) in which we can deploy the apps later on. Our idea is to spin up temporary test infrastructures on-demand, run some tests, and destroy them again after all tests have been completed. When spinning up a test infrastructure we do not want to build again the apps but just deploy the already published apps into the created infrastructure via a dedicated deploy job per app. This deploy job shall be triggered from the upstream pipeline and in the trigger we want to pass the environment details / credentials for the created test infrastructure.

As we're using Kubernetes every app has an associated Helm chart (i.e. deployment definition of our app) which we want to keep as close as possible to the app source code hence we're planning to host them in the same GitLab project which means also that the deploy job must be defined inside of that project. Furthermore the deploy job might vary from app to app (e.g. one app might be a Azure Function App that has a completely different deployment process and won't deploy into the K8s cluster).

I'm aware that there are rules like only: pipeline which I can define for each job but they're not generic enough. If a project pipeline gets triggered by different upstream pipelines, manual triggers, remote triggers etc. it feels very sluggish to mark each job with such a rule. It's also difficult for others to understand which kind of "workflows" have been implemented depending on the trigger type. Also you can define only one "workflow" per trigger type with a fixed stages order among all "workflows". I hope you get what I mean :)

Proposal

Ideally it would be possible to define within one .gitlab-ci.yml multiple CI/CD stage groups that can be individually triggered via a commit, upstream pipeline, trigger and so on. By doing so the readability and understandability of complex pipelines would be greatly improved as you can explicitly declare them in a top level code block.

stages:
  default:
    - build
    - test
    - publish
  deploy:
    - deploy

If there are no stage groups

stages:
  default:
    - stage1
    - stage2

would be equivalent to

stages:
  - stage1
  - stage2

Per definition if no stage group gets explicitly set when running a pipeline the default stage group gets used. Not sure if the element shall be called groups: or stages:. Using stages: might be misleading as you might expect that after the default group the deploy group gets executed in the example above due to the current semantic of stages:.

Triggering a stage group from an upstream pipeline could be possible via

curl -s -X POST \
    -F token=$CI_JOB_TOKEN \
    -F ref=master \
    -F group=$STAGE_GROUP_NAME \
    https://gitlab.com/api/v4/projects/${PROJECT_ID}/trigger/pipeline

That way the .gitlab-ci.yml of the apps could be written as

image: alpine

stages:
  default:
    - build
    - test
    - publish
  deploy:
    - deploy

build:
  stage: build
  script:
    - echo "build"

test:
  stage: test
  script:
    - echo "test"

publish:
  stage: publish
  script:
    - echo "publish"

deploy:
  stage: deploy
  environment:
    name: $DEPLOY_ENV
  script:
    - echo "deploy into env $DEPLOY_ENV"

instead of

image: alpine

stages:
  - build
  - test
  - publish
  - deploy

build:
  stage: build
  script:
    - echo "build"
  except:
    - pipelines
    - triggers

test:
  stage: test
  script:
    - echo "test"
  except:
    - pipelines
    - triggers

publish:
  stage: publish
  script:
    - echo "publish"
  except:
    - pipelines
    - triggers
    
deploy:
  stage: deploy
  environment:
    name: $DEPLOY_ENV
  script:
    - echo "deploy into env $DEPLOY_ENV"
  only:
    - pipelines
    - triggers

which seems to me harder to read and understand what is actually going on, especially if the Yaml grows.