Improve handling of includes in .gitlab-ci.yml to better enable script reuse/templates

Description

Reusing CI/CD process steps is a great practice to help ensure consistency in software delivery, and minimizing the amount of per-job scripting that's needed to write and maintain. We want to offer a flexible, powerful approach for code reuse using templates in this space and this brings us closer to that vision.

Proposal

1. Job-level extends: key-name (Key name does not need to have a dot. We are considering extends taking an array of templates for the next iteration.),
2. Extends does recursive merge,
3. extends: key-name including multi-level

Instead of using a preprocessor-like syntax, we'll use extends as an alternative to anchors, whereupon it does a recursive merge (normal anchors do a shallow merge / replacement.) It happens to work across includes, but has a negative that it works only on top-level keys, but this is not fully defined yet how we and were we are going to support extends.

extends actually adds value even if you are not using include and it solves include preprocessor problem, because it makes it possible to avoid using anchors.

Proposal will work as:

.something: &ref
   ...

job: *ref

job2:
  <<: *ref

where you simply get:

.something:
   ...

job:
  extends: .something

job2:
  extends: job

The extends: will work across includes.

Issue with YAML Anchoring

GitLab 10.5 added the include keyword for including external YAML files in .gitlab-ci.yml files. This can reduce repetition by allowing common jobs and other keywords to be shared across multiple repositories. However, it can be difficult to refactor jobs that have small customizations in different repositories. There is a problem, though, with using anchors across includes.

In my repositories, I commonly have bundle jobs that run docker-compose config to produce a stable Compose file that will be used for various deployments. The bundle share a common base that is defined as a YAML anchor. He's a pared down example.

.bundle_job: &bundle_job
  image:
    name: docker/compose:1.19.0
    entrypoint: [""]
  services:
    - name: docker:stable-dind
  script:
    - docker-compose config --resolve-image-digests > $STACK_NAME.yml
  artifacts:
    name: ${STACK_NAME}-stack
    paths:
      - $STACK_NAME.yml

Concrete jobs (without a leading .) then alias bundle_job to set custom variables or artifacts. Here is an example.

bundle:onboard:
  <<: *bundle_job
  variables:
    EXTERNAL_TRAEFIK_NETWORK: 'true'
    STACK_NAME: $CI_PROJECT_NAME-onboard
    ENVIRONMENT_HOST: $CI_PROJECT_NAME-onboard.$AUTO_DEVOPS_DOMAIN