3 CI Syntaxes: Single Runner/Multiple Stages; Parallel/Sequential; Acyclic-Directed
Pipelines are awesome. You can do multiple jobs, spread out over multiple stages, with stages having lots of jobs run in parallel. Wit this you can cover a lot of use cases. But not all of them. Also, it's sometimes a bit hard to get started because each job starts as a clean slate (by default). So people have asked both for simpler, and more complicated use cases. It's hard to have a single configuration syntax that works for this breadth of usage. But the underlying runner architecture can handle it all just fine. One option to consider is to have 3 different configuration syntaxes, related, but with differences.
- Single runner, multiple stages
- Parallel/sequential pipeline
- Acyclic directed graph: https://gitlab.com/gitlab-org/gitlab-ce/issues/29347
e.g. by specifying a version number or some other selector, you can pick between these three syntaxes.
For (1), all of your jobs would run on a single runner, without needing to pass cache or artifacts between jobs. For (3), each job would need to declare which jobs it depends on, and the system would run each as soon as possible, and only block when explicitly needing to wait for another job to finish. This allows optimal organization of complex build, test, deploy needs.
The idea is that you'd start out at the simplest configuration, then graduate to the more complex level when you're ready, but then also have to deal with the new requirements at that level. E.g. going from (1) to (2) would require adding scripts to recover from failed or missing caches in each job that needs it. Going from (2) to (3) would require declaring job dependencies rather than simpler stage mappings.
Just thinking out loud.
/cc @ayufan @bikebilly