Add provisioner support for ops-gitlab-net deployment pipeline environment

Current state

Provisioner creates deploy projects in https://gitlab.com/gitlab-com/gl-infra/platform/runway/deployments. Each deploy project can run a CI pipeline to deploy Runway services in a specified environment (staging/production).

This runs entirely in gitlab.com. In simple terms, the current deployment flow is: canonical service project pipeline --> canonical deployment project pipeline.

Proposed state

In order to accomplish #230 (closed), the Provisioner needs to support a ops.gitlab.net-only deployment flow of ops service project --> ops deployment project.

Note that ops service project pipeline --> ops deployment project pipeline is a minimal requirement for a deployment. The service owner is responsible for dictating how the ops service project's CI pipeline is triggered (canonical mirror or directly merging/pushing to the ops service project).

Getting there

• [Provisioner] inventory.json is the current source of truth for Runway service definition. We will need a new key (e.g. deploy_in_ops) for each runway service to identify if the service needs an ops deployment pipeline. (https://gitlab.com/gitlab-com/gl-infra/platform/runway/provisioner/-/merge_requests/347)
  • [Provisioner] Provisioner create a ops deployment project in gitlab-com/gl-infra/platform/runway/deployments on ops.gitlab.net for the desired Runway workload in addition to a canonical deployment project.
  • [Provisioner] Provisioner to define a RUNWAY_CI_DEPLOY_ENV project variable to determine if the deployment should run on the canonical or ops project.
• [Provisioner] Add CI job safeguard to ensure changes in deploy environments are flagged. (https://gitlab.com/gitlab-com/gl-infra/platform/runway/provisioner/-/merge_requests/349)
• [Provisioner] Update script to work for both canonical and ops deployment projects (https://gitlab.com/gitlab-com/gl-infra/platform/runway/provisioner/-/merge_requests/348)
• [infra-mgmt] provision gl-infra/platforms/runway/deployments group (https://gitlab.com/gitlab-com/gl-infra/infra-mgmt/-/merge_requests/981) and create group access token (https://gitlab.com/gitlab-com/gl-infra/infra-mgmt/-/merge_requests/980).

Considerations

Deployment projects use gitlab-managed terraform states. When enabling deployment in ops for existing Runway workloads, we need to be careful to perform the migration steps

Note: This is not a problem for newly onboarded Runway services since the terraform state is empty.

We need to account for all state transitions:

1. Workload exists. Canonical -> ops
2. Workload exists. Ops -> canonical
3. New workload. Canonical deployment.
4. New workload. Ops deployment.

No issues: Transition 1 is the what topology service will encounter. Transition 3 and 4 are simple since users start off from a fresh state.

Transition 2 is tricky since that would mean the ops gitlab project should not be removed/archived until the terraform state as been migrated.

Closing summary

This is the high-level overview of the gitlab project setup to deploy from ops.

The planned changes to provisioner are completed:

• Provisioner now creates the ops deployment project and other relevant resources (ops reconciler service account and sets roles).
• CI jobs have been added to flag environment changes
• existing scripts to check for workload removals have been updated to be compatible with ops.

The migration procedure is detailed in #325 (closed) and can be found in docs once #363 (closed) concludes.