Establish deployment pipeline environment migration procedure

Context: #230 (closed)

The goal of this issue is to discuss and establish a set of procedure for migrating such deployment pipeline environments.

Runway deployment projects use gitlab-managed terraform states. When enabling deployment in ops for existing Runway workloads (i.e. already deploying on the canonical deployment project, we need to ensure that the ops deployment project has the updated terraforms state before deployment can begin.

Draft of procedure:

  1. Halt any deployments on both deployment projects. This can be accomplished through runwayctl outlined in #324 (closed).
  2. Migrate the terraform state from the canonical deployment project to the ops deployment project (https://docs.gitlab.com/ee/user/infrastructure/iac/terraform_state.html). This can be done using a script or a set of commands executed by an SRE.

Setup envvars

CANONICAL_PROJECT_ID="<gitlab-project-id>"
CANONICAL_TF_USERNAME="<gitlab-username>"
CANONICAL_TF_PASSWORD="<gitlab-personal-access-token>"
CANONICAL_TF_ADDRESS="https://gitlab.com/api/v4/projects/${CANONICAL_PROJECT_ID}/terraform/state/old-state-name"

OPS_PROJECT_ID="<gitlab-project-id>"
OPS_TF_USERNAME="<gitlab-username>"
OPS_TF_PASSWORD="<gitlab-personal-access-token>"
OPS_TF_ADDRESS="https://ops.gitlab.net/api/v4/projects/${OPS_PROJECT_ID}/terraform/state/old-state-name"

TF_ADDRESS=$CANONICAL_TF_ADDRESS
TF_USERNAME=$CANONICAL_TF_USERNAME
TF_PASSWORD=$CANONICAL_TF_PASSWORD

Init state

terraform init \
  -backend-config=address=${TF_ADDRESS} \
  -backend-config=lock_address=${TF_ADDRESS}/lock \
  -backend-config=unlock_address=${TF_ADDRESS}/lock \
  -backend-config=username=${TF_USERNAME} \
  -backend-config=password=${TF_PASSWORD} \
  -backend-config=lock_method=POST \
  -backend-config=unlock_method=DELETE \
  -backend-config=retry_wait_min=5

Migrate state

TF_ADDRESS=$OPS_TF_ADDRESS
TF_USERNAME=$OPS_TF_USERNAME
TF_PASSWORD=$OPS_TF_PASSWORD

terraform init \
  -migrate-state \
  -backend-config=address=${TF_ADDRESS} \
  -backend-config=lock_address=${TF_ADDRESS}/lock \
  -backend-config=unlock_address=${TF_ADDRESS}/lock \
  -backend-config=username=${TF_USERNAME} \
  -backend-config=password=${TF_PASSWORD} \
  -backend-config=lock_method=POST \
  -backend-config=unlock_method=DELETE \
  -backend-config=retry_wait_min=5
  1. Resume deployments on the ops project.

Future improvements

We can convert it into a helper script and run it as part of the provisioner's deployment. However, we should only do that after the procedure is establish and well-tested.

Edited by Sylvester Chin