Dynamic jobs between stages

Description

From customer:

We require a way to dynamically create jobs in different stages. We are developing our applications in C++ and we require this feature in updating our libraries. Let's say we update library A, depending on how the version of library A was changed we have to decide which dependent libraries should be updated (Please refer to attached png file).

In our current CI configuration we clone all dependent libraries, build them, test them and upload them to our binary repository. The problem is that we have around 50 libraries and some pipelines take hours to finish because the build process is sequential and not parallel. To parallelize the build we need to know which libraries require an update but this list of libraries is not static and it is not convenient to modify the gitlab-ci.yml after each update. We want to have a feature to dynamically modify jobs that should run in the next stage of the current pipeline.

Proposal

image002 gitlab-feature-req

Links / references

Customer issue:

ZD: https://gitlab.zendesk.com/agent/tickets/79007

Documentation blurb

Overview

What is it?

  • Dynamic stages in GitLab-CI

Why should someone use this feature?

  • To reduce CI build time

What is the underlying (business) problem?

  • Many hours are spent rebuilding libraries that haven't updated

How do you use this feature?

  • Create a list of dependencies and allow conditionals (possibly: Only, When) to trigger dependency updates.

Use cases

Repositories with large numbers of dependencies that don't always need to be updated / part of the build.

Feature checklist

Make sure these are completed before closing the issue, with a link to the relevant commit.