Skip to content

Add Google Cloud Compute client

Pedro Pombeiro requested to merge pedropombeiro/439569/add-google-cloud-compute-client into master

What does this MR do and why?

This MR adds a Google Cloud Compute client capable of authenticating through Workload Identity Federation and exposing 3 methods: regions, zones, and machine_types. These methods will be used in #438315 (closed) to expose a runnerCloudProvisioningOptions GraphQL query.

A similar client has been merged recently to communicate with the GCP Artifact Registry: !142289 (merged)

EE: true

Closes #439569 (closed)

There is an opportunity to share code between the 2 client classes derived from GoogleCloudPlatform::BaseClient (for instance, to pass the wlif claim), but that can be left for a follow-up.

🚥 MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

✅

🖼 Screenshots or screen recordings

It's an API class so we don't really have an UI. To see the class in action, see the next action.

⚙ How to set up and validate locally

Although the client class itself is quite simple to use, the setup of the supporting components can be quite involved.

We have two way to set up things: either using the gcp demo project (if you have access) or using a service account.

1️⃣ Set up using the gcp demo project

⚠ This is only available to members with access to the gcp demo project.

  1. In Cloud Run, there is a glgo instance running. Click on the details and copy the url.
  2. In ee/lib/google_cloud_platform/base_client.rb, replace the GLGO_BASE_URL constant with the url from (1.).
  3. In ee/lib/google_cloud_platform/jwt.rb, return a fixed string of your choice for #issuer. I used http://pedropombeiro.gdk.test:3000.
  4. Start your local GDK, access /oauth/discovery/keys and paste the content on a Gitlab.com snippet. Copy the url of the raw form of the snippet.
  5. In Cloud Run, create a new version to deploy and update the GLGO_KNOWN_ISSUERS env variable with the following string: ,<issuer string>;<url of the raw form of the snippet>

Don't forget to set up a Workload Identity Federation properly and get its url without the protocol.

2️⃣ Set up using a service account

  1. Create a service account that has the Compute Viewer role.
  2. Create a json file credentials and download it.
  3. In ee/lib/google_cloud_platform/compute/client.rb, in the #external_credentials function. Replace the method content with the path to the credentials file.

3️⃣ The client class in action

One last setup, there is a guard to make sure that the client class is used in the saas instance only. In ee/lib/google_cloud_platform/compute/client.rb, comment L133.

Now, that the set up is out of the way, let's play! 🕹

One note on the client instance. It is meant to be built and used right away. The different tokens in play here have quite short life times. If you build the client object and call one of the methods a few minutes later, you will probably get the following error GoogleCloudPlatform::Compute::Client::ApiError: 14:Getting metadata from plugin failed with error: #<RuntimeError:"Unable to retrieve Identity Pool subject token {\"error\":\"failed to parse and validate input token: \\\"exp\\\" not satisfied\"}\n">}. That's basically saying that the token expired.

That's ok, how this client class is meant to be used is really to have the client initialization and the function call in a quick succession.

In a rails console:

client = GoogleCloudPlatform::Compute::Client.new(
  project: Project.first, 
  user: User.first, 
  gcp_project_id: '<google project id>', 
  gcp_wlif: '<workload identity federation url WITHOUT the protocol. Eg. start with //iam.googleapis.com/projects>'
)

client.machine_types(zone: 'africa-south1-a')
=> <Google::Cloud::Compute::V1::MachineTypeList: id: "projects/dev-gcp-s3c-integrati-9abafed1/zones/africa-south1-a/machineTypes", items: [<Google::Cloud::Compute::V1::MachineType: accelerators: [], creation_timestamp: "1969-12-31T16:00:00.000-08:00", description: "Efficient Instance, 16 vCPUs, 16 GB RAM", guest_cpus: 16, id: 337016, image_space_gb: 0, is_shared_cpu: false, kind: "compute#machineType", maximum_persistent_disks: 128, maximum_persistent_disks_size_gb: 263168, memory_mb: 16384, name: "e2-highcpu-16", scratch_disks: [], self_link: "https://www.googleapis.com/compute/v1/projects/dev-gcp-s3c-integrati-9abafed1/zones/africa-south1-a/machineTypes/e2-highcpu-16", zone: "africa-south1-a">, <Google::Cloud::Compute::V1::MachineType: accelerators: [], creation_timestamp: "1969-12-31T16:00:00.000-08:00", description: "Efficient Instance, 2 vCPUs, 2 GB RAM", guest_cpus: 2, id: 337002, image_space_gb: 0, is_shared_cpu: false, kind: "compute#machineType", maximum_persistent_disks: 128, maximum_persistent_disks_size_gb: 263168, memory_mb: 2048, name: "e2-highcpu-2", scratch_disks: [], self_link: "https://www.googleapis.com/compute/v1/projects/dev-gcp-s3c-integrati-9abafed1/zones/africa-south1-a/machineTypes/e2-highcpu-2", zone: "africa-south1-a">, <Google::Cloud::Compute::V1::MachineType: accelerators: [], creation_timestamp: "1969-12-31T16:00:00.000-08:00", description: "Efficient Instance, 32 vCPUs, 32 GB RAM", guest_cpus: 32, id: 337032, image_space_gb: 0, is_shared_cpu: false, kind: "compute#machineType", maximum_persistent_disks: 128, maximum_persistent_disks_size_gb: 524288, memory_mb: 32768, name: "e2-highcpu-32", ...

client.regions
=> <Google::Cloud::Compute::V1::RegionList: id: "projects/dev-gcp-s3c-integrati-9abafed1/regions", items: [<Google::Cloud::Compute::V1::Region: creation_timestamp: "1969-12-31T16:00:00.000-08:00", description: "africa-south1", id: 1610, kind: "compute#region", name: "africa-south1", quotas: [<Google::Cloud::Compute::V1::Quota: limit: 500.0, metric: "CPUS", usage: 0.0>, <Google::Cloud::Compute::V1::Quota: limit: 204800.0, metric: "DISKS_TOTAL_GB", usage: 0.0>, <Google::Cloud::Compute::V1::Quota: limit: 575.0, metric: "STATIC_ADDRESSES", usage: 0.0>, <Google::Cloud::Compute::V1::Quota: limit: 575.0, metric: "IN_USE_ADDRESSES", usage: 0.0>, <Google::Cloud::Compute::V1::Quota: limit: 81920.0, metric: "SSD_TOTAL_GB", usage: 0.0>, <Google::Cloud::Compute::V1::Quota: limit: 2500.0, metric: "INSTANCE_TEMPLATES", usage: 0.0>, <Google::Cloud::Compute::V1::Quota: limit: 9.223372036854776e+18, metric: "LOCAL_SSD_TOTAL_GB", usage: 0.0>, <Google::Cloud::Compute::V1::Quota: limit: 100.0, metric: "INSTANCE_GROUPS", usage: 0.0>, 

client.zones
=> <Google::Cloud::Compute::V1::ZoneList: id: "projects/dev-gcp-s3c-integrati-9abafed1/zones", items: [<Google::Cloud::Compute::V1::Zone: available_cpu_platforms: ["Intel Broadwell", "Intel Cascade Lake", "Intel Emerald Rapids", "AMD Genoa", "Intel Haswell", "Intel Ice Lake", "Intel Ivy Bridge", "AMD Milan", "AMD Rome", "Intel Sandy Bridge", "Intel Sapphire Rapids", "Intel Skylake"], creation_timestamp: "1969-12-31T16:00:00.000-08:00", description: "us-east1-b", id: 2231, kind: "compute#zone", name: "us-east1-b", region: "https://www.googleapis.com/compute/v1/projects/dev-gcp-s3c-integrati-9abafed1/regions/us-east1", self_link: "https://www.googleapis.com/compute/v1/projects/dev-gcp-s3c-integrati-9abafed1/zones/us-east1-b", status: "UP", supports_pzs: false>, <Google::Cloud::Compute::V1::Zone: available_cpu_platforms: ["Intel Broadwell", "Intel Cascade Lake", "Intel Emerald Rapids", "AMD Genoa", "Intel Haswell", "Intel Ice Lake", "Intel Ivy Bridge", "AMD Milan", "AMD Rome", "Intel Sandy Bridge", "Intel Sapphire Rapids", "Intel Skylake"], creation_timestamp: "1969-12-31T16:00:00.000-08:00", description: "us-east1-c", id: 2233, kind: "compute#zone", name: "us-east1-c", region: "https://www.googleapis.com/compute/v1/projects/dev-gcp-s3c-integrati-9abafed1/regions/us-east1", self_link: "https://www.googleapis.com/compute/v1/projects/dev-gcp-s3c-integrati-9abafed1/zones/us-east1-c", status: "UP",

You can also try exceptions:

client = GoogleCloudPlatform::Compute::Client.new(
  project: Project.first, 
  user: User.first, 
  gcp_project_id: '<google project id>', 
  gcp_wlif: '<workload identity federation url WITHOUT the protocol. Eg. start with //iam.googleapis.com/projects>'
)
GoogleCloudPlatform::Compute::Client::ApiError: 3:Request contains an invalid argument.. debug_error_string:{...}
Edited by Pedro Pombeiro

Merge request reports