Skip to content

Add initial support for DAG

Kamil Trzciński requested to merge implement-dag into master

What does this MR do?

This adds a DAG support for GitLab CI. This works.

Design

It implements the simple extension to CI job:

build:
  stage: build
  script: echo Hello World

setupenv:
  stage: build

rspec:
  stage: test
  needs: [build]

This makes the rspec to run right after the build finishes regardless of status of other jobs in stage: build.

The logic of needs:

  1. If defined, the dependencies: needs to be subset of needs:,
  2. If defined, and the dependencies: are not defined, the job will depend by default on all jobs that are defined in needs: only,
  3. If defined, runs the pipeline out of order,
  4. If defined, the needs: need to depend on a job from prior stages,
  5. If defined, if the needed job is a parallel, it is expanded to need all parallel jobs,
  6. If not defined, it runs pipeline as with stages (backward compatibility),
  7. The needs: is opt-in approach, by default pipeline runs in stages, if needs is defined it changes the behaviour of just single job,
  8. needs: [] is invalid and is todays limitation, so you cannot start the job in second stage with needs: []

Notes

  1. It is quite performant, we validate dependent jobs only on finish of the job that this depends on,
  2. It is backend only, no frontend,
  3. Actually, this is pretty complete, except tests,
  4. It should cover every aspect I think: a. parallel, b. validation of dependencies, c. ensure that needs has to be from prior stages, etc.
  5. The processing part of DAG is currently disabled, and can be enabled with ci_dag_support,
  6. The DAG processing is extension to current stages processing, once job finishes it is validated if it affects other jobs, and other jobs can be started then,

Resolves https://gitlab.com/gitlab-org/gitlab-ce/issues/47063

Does this MR meet the acceptance criteria?

Conformity

Performance and testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team
Edited by Kamil Trzciński

Merge request reports