Self-modifying/extending pipelines MVC

Problem to Solve

There are certain kinds of pipelines that truly need dynamic behavior in order to be implemented in an efficient way. For example, an operation (DAST & SAST are good examples) where a previously unknown number of jobs should be spawned in order to maximize parallelization. We can solve this by making it possible for a job to spawn other jobs in the same pipeline, resulting in a dynamic CI/CD pipeline.

Use Cases

DAST & SAST

Because one particular probe can be relevant or not depending on what's been detecting in the project, we would have a detection job responsible for spawning the probe jobs. It's important to make this dynamic since what's detected can change from one commit to another; we can't generate the content of .gitlab-ci.yml based on what's detected in one particular state. See https://gitlab.com/gitlab-org/gitlab-ee/issues/5232 for additional information.

Proposal

Introduce a new kind of job artifact that would be parsed and turned into extra jobs (database records describing pending jobs). The jobs would be added to an existing stage of the pipeline. The targeted stages are ahead of the current stage.

Warning! We need to implement some restrictions to prevent DDoS attacks.

Links / references

See https://gitlab.com/gitlab-org/gitlab-ee/issues/5232 for use case.