Skip to content

Allow jobs in later staged to be started manually if all their dependencies are available

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Proposal

In large pipelines (with limited number of runner executors), some jobs are grouped together in stages to avoid having too many runners to be processing the same pipeline. All those jobs have very limited dependencies that are available way before their stage is reached and they can start. However, sometimes it would help productivity to start the execution of one (or many) of this job manually to ensure there won't be any error for instance.

Here is an example of .gitlab-ci.yml to illustrate the feature:

stages:
  - build
  - tests1
  - tests2
  - tests3

build-project:
  stage: build
  script: mvn clean package

tests1_1:
  stage: tests1
  script: sleep 1800
  dependencies: [build-project]
  allow_failure: true

tests1_2:
  stage: tests1
  script: echo "Still another time-consuming job 1_2"
  dependencies: [build-project]
  allow_failure: true

tests2_1:
  stage: tests2
  script: echo "Still another time-consuming job 2_1"
  dependencies: [build-project]
  allow_failure: true

tests2_2:
  stage: tests2
  script: echo "Still another time-consuming job 2_2"
  dependencies: [build-project]
  allow_failure: true

tests3_1:
  stage: tests3
  script: echo "Yet the last time-consuming test"
  dependencies: [build-project]
  allow_failure: true

One can see that there is no manual job (and that is normal). For the sake of simplicity, there is only few jobs in the example above, but imagine dozen of jobs per stage. Actually, the job tests3_1 will start after all the jobs in tests2 stage have completed which is the correct behavior.

The aim of the feature is to be able to manually start the job tests3_1 once the build-project job is complete (since it is the only dependency). Note that we DO NOT want to add when: manual because then it would be necessary to start it manually and most of the time, it is wanted that the job simply starts without any action when the previous stage has finished. This also differ from needs: [...] because then all the jobs would start, taking up all the runners (which is exactly what is to be avoided since otherwise all the jobs would have been in the same stage).

There are many useful use cases I can see for this feature to be implemented:

  • there is a lot of runners available (for instance during week-end), so one can start multiple jobs manually to speed up the pipeline build
  • one can start a job manually to see if it will fail later (and work on a fix even before the pipeline fails)
  • avoid feeding up all the runners for a single pipeline, while still being able to make it complete faster by starting jobs manually

A far as I can tell, I don't see big changes for this: in the pipline view, instead of showing a "Created" icon (which cannot be clicked), just display the "Play" icon when all the dependencies are available. Also that feature will be directly available without any need to change the CI/CD configuration.

Any feedback is welcome, thanks !

Edited by 🤖 GitLab Bot 🤖