Move cluster creation under Terraform

Problem to solve

As a Platform Engineer / DevOps Engineer, I want to manage my cluster creation process as code, so that everything is under version control.

As a CTO, I want to use a tool that supports various providers and is easily extendable with new providers.

Intended users

@nagyv-gitlab run dozens of interviews around Kubernetes and Terraform usage in the past months, and every participant uses Terraform to set up and tear down their clusters, and related cloud project and resources.

User experience goal

The user should be able to create and own a cluster in a transparent way using GitLab
The user should be able to edit the cluster provided by GitLab

Proposal

MVC

Provide a project template to create a GKE and AWS cluster as they are provisioned today with GitLab
- use the GitLab backend for Terraform
The provided template should attach the created cluster to a project given to it in a CI variable
- the backend file should be derived from the project we attach the cluster to, this way a single project can be used for several clusters
- the current user becomes an owner of the project
Create a UI to set up a new project with the selected template, and all the necessary env variables needed to run the cluster provisioning step
Switch the current UI to the new UI
- should we provide an infrastructure project to existing users or not?

The resulting Terraform repo should be (over)documented.

It should contain a README,
the CI should describe every step and env var,
all Terraform files should describe what a given resource is for and what our initial setup is

Current UI based cluster actions

All of these can stay on the UI and be used as they are today.

set / unset GitLab integration
set / unset GitLab managed cluster
install applications
specify a cluster management project
clear the cache
remove integration
remove integration and resources

Cluster configuration project path

How can the user set the cluster configuration project's "path"?

we provide two inputs:
- select from the user's groups
- text input to provide the project's name
- for project- and group level clusters the select is preselected for the project's group
- for project level clusters the text input is prefilled with <project-name>-cluster
- for group level clusters the text input is prefilled with <group-name>-cluster
- if the pre-filled project exists, we use a counter <project-name>-cluster-1

Removing the cluster

What happens if the user removes the cluster using the "remove integration" button?

we keep the cluster configuration project

Removing the cluster configuration project

What happens if the user removes the cluster configuration project?

we keep the cluster and its imported state

Making the setup user-friendly

Is there a nice way to get Google's OAuth work with such a setup or we'd need the user to create a service account for us to use? Or our script would use OAuth to create a service account, and attach that to the new project?

Edge cases

What happens if the user has no Runner attached or run out of Runner minutes?

On the "form' page, we should check if runners are available for the user, and put a huge red notification out if they are not. The notification should state that clusters are created via the runner, they have no runner available, they can still proceed, and have all their code ready, but the cluster won't be created, and its pipeline should be started manually, once runners will be available.

What happens if the user removes the Terraform state file, then re-runs the Terraform project?

We would create a new cluster, and attach it to GitLab beside the other cluster.

Can a single Terraform project be used to create multiple clusters and attach them to different resources, levels?

If we want to support this use-case, then we should provide a user to choose from existing repos when creates a new cluster
we might support this use-case without a UI. As the details where to import the cluster come from env vars, the user can set up a job that uses different variables.

Later steps

provide a UI to configure the cluster at creation as described in #29862 (comment 215822555)

Further details

@bdowney has great repos for EKS, GKE, AKS under https://gitlab.com/gitops-demo/infra

The created Terraform project might be configure using input variables. It would be great to iterate towards a "Terraform input UI" where users can run any Terraform module with one-off parameters provided through a UI. Basically, the UI would set TF_ENV_ variables, and the core feature is to auto-generate the UI based on code inspection.

Permissions and Security

Documentation