Allow pipelines to be populated async

Problem

Pipeline creation of gitlab-org/gitlab via the Web request times out more often. This is because the pipeline creation takes more than the request timeout. Creating complex pipelines with many rules can take longer and longer the more jobs are added to it.

Idea

One idea we could explore, which could be the predecessor of Add support for extending existing pipelines wi... (#393334) is to have a way to create an empty pipeline and then add jobs asynchronously in batch:

  1. Trigger pipeline creation - e.g. Click "run pipeline" button.
  2. The pipeline creation logic starts synchronously during the request (parsing CI configs and evaluating initial conditions).
  3. If the pipeline is filtered out by workflow:rules (recommended to have) we would not persist the pipeline and finish.
  4. Otherwise, persist an empty pipeline and add jobs asynchronously in batches.
  5. For each batch of jobs we evaluate rules/except/only conditions and add or not jobs to the pipeline.
  6. After iterating through all batches, we continue the pipeline creation logic and finally start the pipeline processing.

Pros:

  • Pipeline creation could be improved drastically. Perceived performance matters a lot as usually developers are blocked waiting for a persisted pipeline in order to run "merge when pipeline succeeds".
  • We gain a lot of headroom so we can resolve the problem of request timeouts.
  • We can build internal APIs to add jobs to an existing pipeline (especially in bulk) and reuse this abstraction for Add support for extending existing pipelines wi... (#393334)

Cons:

  • Risk in creating empty pipelines (in skipped status) when before we would not create a pipeline at all. The risk can be largely mitigated using workflow:rules. Some edge cases can slip through.
  • There might be some behavior in the code built with assumptions that a valid pipeline always has jobs.