Skip to content

Use gcs as remote backend for deployment projects

As a follow up to #230 (closed), having deployment projects use gcs instead of gitlab to manage the terraform state would allow both .com and ops environments to access the state file.

This gives us the following benefits

  1. smoother migration path (we can toggle the CI yml to cutover reconciler deploy from .com to ops)
  2. .com pipeline can run plan, allowing users to iterate without having ops access. The .com pipeline can work with a readonly roleset too.

Implementation ideas

  1. Configure using a .tfbackend file (https://developer.hashicorp.com/terraform/language/settings/backends/configuration#file) and switch between http.tfbackend and gcs.tfbackend so that other runway deployment projects can continue using http backend.
  2. Create bucket in provisioner within the runway gcp project in which the service is deployed.

Some complexities to deal with

  1. we cannot use gitlab-terraform since that is meant to target gitlab as a http remote backend. We will need to switch between gitlab-terraform and terraform depending on http/gcs backend.
  2. .tfbackend files cannot configure the type of backend (e.g. http or gcs), only the configuration values IIUC. runwayctl will have to do so file manipulation to write the correct backend.tf file during init. I fear this could break the destroy workflow since we have a bootstrap.tf

In https://developer.hashicorp.com/terraform/language/settings/backends/configuration#partial-configuration

When using partial configuration, Terraform requires at a minimum that an empty backend configuration is specified in one of the root Terraform configuration files, to specify the backend type

Note: this is considered a good-to-have and should be prioritised lower than other reliability improvement issues.

/cc @gsgl thanks for the great idea, please update as required 👍

update: dropping this issue from the &1348 epic since it is not a hard requirement for decoupling deployment from .com

Edited by Sylvester Chin