Allow for indicating that a predecessor job is optional in a DAG

Release post

With the implementation of our directed acyclic graph (DAG) in GitLab CI we introduce a syntax that makes a job run only if a different job finish its run (without waiting to its own stage). however if the DAG syntax is combined with rules syntax - which detairmine if a job will gets added to a pipeline there can be a situation where a job is waiting for a job that does not exists in the pipeline (because it did not got added to the pipeline), this will cause the pipeline to fail only on run. In this release we are adding the optional keyword to any DAG job which will process the pipline with the condition that If the optional job exists, then wait for it, otherwise run as part of the chosen stage, this will allow you to saifly combined the very popular rules syntax with the growing popularity of the DAG concept

Problem to solve

We are implementing an MVC for a directed acyclic graph (DAG) execution model in GitLab CI via https://gitlab.com/gitlab-org/gitlab-ce/issues/47063#note_198535521, which introduces the following syntax for marking a job as running after another job, without waiting for its own stage:

deploy:
  stage: deploy
  needs: [test]
  ...

A "smart default" we chose was for anything indicated as needed by a job was in fact needed, and if the job did not exist (likely because it didn't match to the pipeline's only/except rules), that it would also not run. However, there are some unusual cases where the intent would be:

If job _x_ exists, then wait for it, otherwise run as part of the chosen stage

This is not possible with the current syntax, and this limits usefulness of a whole DAG feature significantly. Due to other limitations teams are often forced to duplicate jobs, this in turn causes snowball effect for DAG, where there is currently no way to express needs: [A or B]. Allowing to specify job in needs which might not exist in the given pipeline makes gitlab-ci.yml files more concise.

Intended users

Further details

Proposal

We can make this easier by allowing you to specify that the job should go ahead if the job it refers to does not exist:

job1:
  needs: [a, b, c, { job: d, optional: true }, e, f]

The default value, when not explicitly overridden, is false (which is the current behavior today).

Note that this should work in concert with #233876 in cases where both are provided.

Permissions and Security

Documentation

Testing

What does success look like, and how can we measure that?

Links / references

This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.