Add support for extending existing pipelines with dynamically generated jobs
Description
In Make it possible to use any language to generat... (#22060 - closed) I suggested building a feature that will allow users to dynamically generate a pipeline using their programming language of choice, by generating .gitlab-ci.yml content that passed in an artifact would be then used by GitLab to create a pipeline. We have been able to build that feature on top of parent-child pipelines feature.
I'm not sure how the adoption of this feature looks like, but after a couple of years of having this feature implemented I rarely hear about users actually using it. While I'm getting engaged by our support team and directly supporting our users and customers I hear that our users are frequently surprised that this feature exists, what makes me think that the adoption might not be as significant as I would it expect to be.
While I may be mistaken (and perhaps there is a better way to measure the adoption than just using my subjective perception of the adoption) I believe that there are great ways to improve the adoption and make the feature even more useful and powerful. Regardless of what the adoption really is and how users are leveraging dynamically generated child-pipelines we may want to improve the workflow a bit.
Proposal
One way to make this feature more powerful is to remove the need of using child pipelines, what was the initial proposal in in Make it possible to use any language to generat... (#22060 - closed). While child pipelines are great, they are also making it a bit more difficult to see what is happening in the CI environment, and introduce additional complexity that may not be needed in case of dynamically generated pipelines.
Would it be possible to just run a seed build that generates .gitlab-ci.yml content, send it back to GitLab and augment existing pipeline with new jobs' definition? This could cover a very common use-case: having just a single seed build to generate entire pipeline dynamically. In such case having a parent-pipeline with just a single job, and a very complex child pipeline makes the whole idea of dynamically generated pipeline a bit too complex.
Augmenting existing pipeline with new configuration might not be trivial, but this can be done and can potentially unlock better adoption of this feature.
Idea 2
One idea we could explore, which could be the predecessor of Add support for extending existing pipelines wi... (#393334) is to have a way to create an empty pipeline and then add jobs asynchronously in batch:
- Trigger pipeline creation - e.g. Click "run pipeline" button.
- The pipeline creation logic starts synchronously during the request (parsing CI configs and evaluating initial conditions).
- If the pipeline is filtered out by
workflow:rules(recommended to have) we would not persist the pipeline and finish. - Otherwise, persist an empty pipeline and add jobs asynchronously in batches.
- For each batch of jobs we evaluate rules/except/only conditions and add or not jobs to the pipeline.
- After iterating through all batches, we continue the pipeline creation logic and finally start the pipeline processing.
This new strategy could be enabled via a project setting or directly in the yaml syntax (e.g. workflow:populate_async:true).
Pros:
- Pipeline creation could be improved drastically. Perceived performance matters a lot as usually developers are blocked waiting for a persisted pipeline in order to run "merge when pipeline succeeds".
- We gain a lot of headroom so we can resolve the problem of request timeouts.
- We can build internal APIs to add jobs to an existing pipeline (especially in bulk) and reuse this abstraction for Add support for extending existing pipelines wi... (#393334)
Cons:
- Risk in creating empty pipelines (in
skippedstatus) when before we would not create a pipeline at all. The risk can be largely mitigated usingworkflow:rules. Some edge cases can slip through. - There might be some behavior in the code built with assumptions that a valid pipeline always has jobs.