Allow Runway services deploying on ops to dry-run on gitlab.com

Originating from #356 (comment 2089450476)

The motivation is for dev workflow on the canonical project to not be disrupted even if the Runway workload is deployed from ops.gitlab.net. This would mean the canonical deployment project would need to be able to run some workloads (e.g. dry runs).

The terraform state should not be stored on the canonical deployment project since it may not be available during an outage on gitlab.com.

Idea 1: Use gcs bucket as remote backend

details in #264 (closed)

Idea 2: Have canonical deployment project point to the terraform state in ops.gitlab.net

Implementation considerations

• runwayctl needs to set 3 envvars if required https://gitlab.com/gitlab-org/terraform-images#required-environment-variables
  • canonical repo needs a ops PrAT instead of using its ci job token (TF_PASSWORD)
  • canonical repo needs to know ops repo project id TF_ADDRESS
• non-active projects can continue to run reconciler plan but not reconciler apply
• keep terraform state in ops.gitlab.net

pros

• runwayctl logic has fewer forks in logic to accommodate a different backend
• there is no need to provision a gcs bucket on the provisioner
• bootstrap.tf works as-is and we do not break deletion flow

cons

• tfstate is dependent on either projects (compared to gcs which is external to gitlab)

Work required for both options

• support oidc exchange on both canonical and ops deployment project
• two sets of service accounts if deploy_from_ops is set to true

---

Summary

We gone ahead with option 2. Validated with #361 (comment 2093467790). The required MRs have been merged and the functionality is available as of runway v3.41.0: runwayctl!614 (comment 2095620088)