API: Implement GraphQL and REST endpoints to return Google Cloud project setup script

Expose a GraphQL field and a REST endpoint that provide the frontend with the required script for setting up a Google Cloud project to use for runner creation:

From @pedroms in Slack (internal-only):

In this step of the runner creation flow:

1. The user provides the Google Cloud project ID where they would like to provision the runner in. This field is empty by default, user must indicate the project every time.
2. That Google Cloud project must be configured to connect to GitLab and use the runner. This configuration is specific to the runner creation use case. When that project is properly configured:
   1. It enables GitLab to fetch the allowed regions/zones/machine types for the next step/screen in this runner creation flow.
   2. It enables GRIT/Terraform to do its magic on that Google Cloud project.
3. The configuration would involve 1+ manual scripts (using glab or gcloud, not sure). I imagined it would (at least) include:
   1. Enable the required APIs in Google Cloud (e.g. Compute Engine API). This ensures that the needed products/services are available.
   2. Grant the necessary IAM roles or permissions to the WLIF principalSet (e.g. roles/compute.instanceAdmin). The ensures that GitLab can connect to the runners in Google Cloud and read/write whatever is needed for this integration (e.g. read allowed machine types for that Google Cloud project).
   3. Maybe create a service account or grant IAM policies to an existing service account (I remember Jason from Google mentioning we needed a service account for something in runners in Google Cloud)
   4. <more?>

changed milestone to %16.10

added Category:Fleet Visibility GraphQL backend devopsverify featureaddition gcp+gitlab integration grouprunner sectionci typefeature labels

assigned to @pedropombeiro

added to epic &12112

marked this issue as related to #439486 (closed)

added workflowplanning breakdown label

mentioned in issue gitlab-runner#30935 (closed)

changed the description

marked this issue as blocked by #438315 (closed)

Update 9 Feb 2024

What's left to be done:

There is a meeting scheduled for next Monday to ensure we're in sync regarding what is to be accomplished in this issue. Development will start as soon as there's agreement.

changed health status to needs attention

changed the description

@pedropombeiro in addition to what I mentioned in Slack (that is currently in the description), these steps may need to be included in that pre-configuration stage — from @sgoldstein's point in this agenda about Google Cloud Infrastructure Manager:

1. Create project
2. Enable Infrastructure Manager API
3. Enable other required APIs (e.g. GCS, GCE, GKE, etc.)
4. Assign service account role with appropriate permissions to Infrastructure Manager
5. Upload terraform template to GCS bucket (or use public git repo)
6. Trigger `terraform apply` via gcloud/api
7. list created resources, view log, etc.

mentioned in epic &12111 (closed)

mentioned in merge request !144454 (merged)

changed the description

mentioned in merge request !144505 (merged)

changed the description

added workflowin dev label and removed workflowplanning breakdown label

mentioned in issue #428498

Update 14 Feb 2024

What's left to be done:

Need to discuss with @ifarkas to start building the script that will be shown to the user. An MR with the backend query scaffolding is already in draft.

MRs

1. GraphQL: Implement projectSetupShellScript (!144505 - merged)

Hi @ifarkas You mentioned you could help with the required commands here. When you have some time, would you mind adding them to the issue description (even if they're not 100% complete)? I can then test with a sample project to make an end-to-end test.

The draft MR is currently returning a markdown document that renders as:

@pedropombeiro, sure! I am not aware of the specific requirements for Runner integration, the example you provided looks good. To confirm the individual items in the description:

1. Enable the required APIs in Google Cloud (e.g. Compute Engine API). This ensures that the needed products/services are available.

To enable services (eg. Compute engine), you can use

$ gcloud services enable compute.googleapis.com

2. Grant the necessary IAM roles or permissions to the WLIF principalSet (e.g. roles/compute.instanceAdmin). The ensures that GitLab can connect to the runners in Google Cloud and read/write whatever is needed for this integration (e.g. read allowed machine types for that Google Cloud project).

Again, this should be adjusted for the Runner integration, but to grant compute.instanceAdmin to developer+ access level:

$ gcloud projects add-iam-policy-binding ifarkas-b0149eb0 \
    --member=principalSet://iam.googleapis.com/projects/173030515811/locations/global/workloadIdentityPools/gitlab-wlif-i/attribute.is_developer/true 
    --role=roles/compute.instanceAdmin

3. Maybe create a service account or grant IAM policies to an existing service account (I remember Jason from Google mentioning we needed a service account for something in runners in Google Cloud)

That also looks good in your example.

Also, the MRs for the existing scripts might be useful for the Runner integration too:

• Google Cloud WLIF creation script to update int... (!144602 - merged)
• Google Cloud integration API: GAR integration s... (!144787 - merged)

Thanks @ifarkas! I guess we could also pipe the scripts from your endpoints through curl, but that might make users nervous.

changed the description

added workflowin review label and removed workflowin dev label

changed health status to on track

changed the description

Update 16 Feb 2024

What's left to be done:

There is still discussion whether the actual script will be provided by the backend or the frontend. In the meantime, I'll be working out the required commands, and testing them on a new GC project.

MRs

1. GraphQL: Implement projectSetupShellScript (!144505 - merged)

changed the description

changed the description

mentioned in merge request !145525 (merged)

changed title from GraphQL: Implement query to return Google Cloud project setup instructions to REST API: Implement endpoint to return Google Cloud project setup script

changed the description

changed title from REST API: Implement endpoint to return Google Cloud project setup script to API: Implement GraphQL and REST endpoints to return Google Cloud project setup script

changed the description

Update 23 Feb 2024

What's left to be done:

Review of MR.

MRs

1. Add runner_deployment_project.sh REST endpoint (!145525 - merged)

added workflowverification label and removed workflowin review label

Verified in staging:

closed

The workflow label was automatically updated to workflowcomplete because you closed the issue while in workflowverification.

If this is not the correct label, please update.

To avoid this message, update the workflow label as you close the issue. This message was generated automatically. You're welcome to improve it.

added workflowcomplete label and removed workflowverification label

mentioned in merge request !146223 (merged)

Update 1 Mar 2024

What's left to be done:

The last MR is being merged as I type this, so this issue will be closed soon.

MRs

1. Minor fixes to the Google Cloud runner provisio... (!146223 - merged)

mentioned in merge request !146894 (merged)

changed epic to &13494 (closed)