Add new CI template using the GitLab Terraform Backend

Problem

We should add a new CI template using the GitLab Terraform Backend in order to simplify it's usage and its associated documentation.

To not break the current Terraform.gitlab-ci.yml we'll create a Terraform.latest.gitlab-ci.yml. The former should then be deprecated on %14.0.

Initially, the expectations we're trying to match with this new template are:

  • We want to promote the use of https://gitlab.com/gitlab-org/terraform-images and the GitLab HTTP backend configure by default.
  • The user should be able to override the terraform image to chose between using the stable tag or pin it to any other tag.
  • The user should have the flexibility to turn-off the GitLab HTTP backend and possibly setup their own backend somewhere else. The extra config for other backends would then be set by the TF_HTTP_* variables
  • The user should be able to configure their terraform state and cache artifact name.
  • The user should be able to configure a Terraform pipeline for multiple environments with a considerably concise .gitlab-ci.yml
  • The user should be able to cherry pick the template jobs that they want to use. For instance, once could want to have init, plan and build but skip validate, or write their own validate job.
  • The user should be able to override the default pipeline stage names.

Release notes

A new GitLab CI template provides GitLab users the ability to have their Terraform pipelines set up using GitLab - Terraform integrations without any manual work. We have seen a lot of interest in the latest GitLab developments around improved Terraform support. The new CI template allows for an effortless getting started with these features.

https://docs.gitlab.com/ee/user/infrastructure/

Default Terraform pipeline

In this scenario, the user accepts all our defaults and uses the GitLab HTTP Backend. A whole terraform pipeline will be available with the following:

# .gitlab-ci.yml

include:
  - template: Terraform.latest.gitlab-ci.yml

variables:
  TF_STATE_NAME: default # if not using GitLab's HTTP backend, remove this line and specify TF_HTTP_* variables
  TF_CACHE_KEY: default

Terraform Pipeline with multiple environments

In this example show:

  • How one could fully customize a stage by overriding name, adding your own scripts or just skipping stages. It happens with the auxiliar template called Terraform/Base.gitlab-ci.yml which will include hidden jobs that are optionally added to the template.
  • How to deploy multiple environments
# .gitlab-ci.yml

include:
  - template: Terraform/Base.gitlab-ci.yml

image: registry.gitlab.com/gitlab-org/terraform-images/branches/mattkasa-tf-state-name-0.13:04f3a1f79dc2099ffc917b42f0fbe8c1754ae257

stages:
  - setup # renaming the default 'init' template to 'setup'. (also adding extra config '.production')
  - lint  # renaming the default 'validate' template to 'lint'. (also adding extra config '.production')
  - plan  # renaming the default 'build' template to 'plan'. (also adding extra config '.production')
  - apply # renaming the default 'deploy' template to 'apply'. (also adding extra config '.production')

# Configures parallel execution in the same pipeline for separate environments.
# To add more environments, just append them to the `matrix:` list as below
.production:
  parallel:
    matrix:
      - TF_ROOT: production/dns             # the directory where you want to execute the terraform commands for such environment
        TF_STATE_NAME: production-dns       # each environment needs its unique `TF_STATE_NAME`
      - TF_ROOT: production/servers
        TF_STATE_NAME: production-servers

setup production:
  extends:
    - .production
    - .init
  stage: setup

lint production:
  extends:
    - .production
    - .validate
  stage: lint

plan production:
  extends:
    - .production
    - .build
  stage: plan

apply production:
  extends:
    - .production
    - .deploy
  stage: apply

/cc @nagyv-gitlab @nicholasklick @Alexand

Edited by Viktor Nagy (GitLab)