Include external files in .gitlab-ci.yml

Description

When companies have many projects, they often have standardized testing and deploying processes, which are often reflected in .gitlab-ci.yml. We should make it easy to share these processes and DRY up CI/CD configuration by letting one project import/include from another project's repo; either its .gitlab-ci.yml or another specified file. By allowing multiple imports, configuration could be componentized with a mix-and-match approach to constructing .gitlab-ci.yml content, possibly using YML templates defined in the included file.

Since this is targeting larger organizations, it will be ~"EE Premium".

Proposal

Add a new include keyword in .gitlab-ci.yml that allows inclusion of external definitions.

The file can be:

• local to the same repository, referenced using the path
• remote in a different location, accessed using HTTP(S) protocol, referenced using the full URL

1. This can be used at the top level of the file, not inside any section or job definition
2. Includes are done first, the resulting file is composed, then it is evaluated as it were a single local file
3. Support for multiple include can be done using an array instead of a single value

Still to be defined / possible improvements

1. Nested use of include (keyword used in included files too)
2. Support for HTTP(S) authentication
3. Relative/absolute local paths
4. Lint support for included files
5. Behavior if used in ~CE or ~"EE Starter"

Examples

Import a definition from another file in the same project:

include: /templates/.gitlab-ci-templates.yml

Import a definition from another project:

include: https://gitlab.com/same-group/another-project/raw/master/.gitlab-ci-templates.yml

Include a specific version of the file:

include: https://gitlab.com/gitlab-org/gitlab-ce/blob/8efdf75b/.gitlab-ci-templates.yml

Use template definitions:

include: https://gitlab.com/same-group/another-project/raw/master/.gitlab-ci-templates.yml

test:
  <<: *predefined_test

Out of scope

Mix-and-match raw (direct import into job):

test1:
  include: same-group/template-repo/.gitlab-ci-rspec.yml

test2:
  include: same-group/template-repo/.gitlab-ci-rubocop.yml

This last option might seem easier because you can ignore YAML templates, which are pretty awkward. But you quickly end up including the same file multiple times, buried deep in the config file, and that feels like an anti-pattern. Dropping this scope may also ensure that each included YAML file can be fully validated on its own.

Links

• Requested here: https://forum.gitlab.com/t/ci-common-configuration/3862
• Older proposal: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/1258

Original proposal

Add keyword `include` which contains a string or array of strings.

Use a fully-defined pipeline from another project:

include: same-group/another-project/.gitlab-ci.yml

Use template definitions:

include: same-group/template-repo/.gitlab-ci.yml

test1:
  <<: *predefined_template

Mix-and-match template definitions:

include:
  - same-group/template-repo/.gitlab-ci-rspec.yml
  - same-group/template-repo/.gitlab-ci-rubocop.yml

test1:
  <<: *rspec_template

test2:
  <<: *rubocop_template

Include from arbitrary file, URL, or specific branch:

include:
  - group/project/directory/file.ext
  - user/project/directory/file.ext
  - https://server/url
  - group/project#branch

Using predefined templates seems like the recommended best practice, but the first way is a great way to get started, or use if a company has a rigid reusable process. It's comparable to including all of Twitter bootstrap vs including only the components you use.