Make it easier for .gitlab-ci.yml to be generated

Description

We have a large number of tests and we like each test to run separately in Gitlab-CI. Since we don't want to update .gitlab-ci.yml every time we add a test, we have a script that generates .gitlab-ci.yml from .gitlab-ci.yml.in. To make sure we don't forget to update .gitlab-ci.yml, we have a test that regenerates it and errors out if it is different.

This is a bit annoying, because:

  1. we still have to manually commit the changes to .gitlab-ci.yml;
  2. .gitlab-ci.yml becomes pretty ugly with all the auto-generated stuff.

Proposal

  1. Add a possibility for .gitlab-ci.yml to include other files.
  2. Make it possible for .gitlab-ci.yml to be modified by one of the pipeline stages.

I'm not entirely sure that the second option is even feasible.

Links / references

Tangentially related: https://gitlab.com/gitlab-org/gitlab-ce/issues/34290

Our .gitlab-ci.yml setup:

Documentation blurb

Overview

What is it? Why should someone use this feature? What is the underlying (business) problem? How do you use this feature?

Use cases

Who is this for? Provide one or more use cases.

Feature checklist

Make sure these are completed before closing the issue, with a link to the relevant commit.