Make it possible to use any language to generate `.gitlab-ci.yml` and pipelines

Description

.gitlab-ci.yml can quickly become very complex. .gitlab-ci.yml in GitLab CE is quite complex already, and we are struggling with maintaining it sometimes.

Proposal

Make it possible to allow anyone to use any language they would like (Python, Ruby) to create CI/CD configuration.

How to do that?

Jenkins made it possible to use seed jobs, and this is fairly established concept we could use too.

We could make it possible to extend existing pipeline with configuration from .gitlab-ci.yml generated by a seed build. It does not matter what technology / language / approach seed build uses, what matters is a valid .gitlab-ci.yml sent back in the artifacts archive.

Imagine following scenario:

1. Define a seed build in .gitlab-ci.yml:

image: ruby:alpine

seed:
  script:
    - gem install gitlab-ci-config
    - gitlab-ci-config generate my-config.rb > artifacts/.gitlab-ci.yml
  artifacts: # ...

gitlab-ci-config is an example, it can by a piece of Ruby code that is not a gem, it can be a Python code, or anything else. We can build one or two tools by ourselves to make it easier to adopt this feature, but it is a secondary aspect.

2. Push code to GitLab

After pushing code with .gitlab-ci.yml like this to GitLab, GitLab Runner would pick the job, to generate .gitlab-ci.yml in a secure way.

It would push artifacts with .gitlab-ci.yml back.

In case of an invalid .gitlab-ci.yml generated, change seed build status to failed.

3. Augment existing pipeline to extend it with subsequent stages

GitLab should be able to easily see that seed build has artifacts with a new .gitlab-ci.yml. We already do something like this with codequality, sast and other jobs.

It would then extend existing pipeline with additional stages / builds, and the pipeline would look like:

seed -> build -> test -> deploy

What do you think @ayufan @bikebilly @markpundsack?

/cc @godfat