Refactor the GCP artifact registry Client

🔭 Context

In Add the GCP technical demo service (!139797 - merged), we added a service to read docker images out of an Google Artifact Registry repository.

At that time, we created a "client" class that would encapsulate everything required (authentication steps + calling the Google Artifact Registry API through a satellite service) and to provide a simple api to get those docker images.

This was done as experimental code for a technical demo.

This MR aims to evolve this client class to have a more permanent version of that part. This refactoring will cover:

• Add support to other Google Artifact Registry APIS: get the details of a repository and get the details of a given docker image.
• Implement an architecture change in the satellite service. See https://gitlab.com/gitlab-org/architecture/gitlab-gcp-integration/glgo/-/issues/13. In simple words:
  • Previously (master):
    • Generate a JWT and call an API on glgo along with that token. glgo would take care of everything, including the token exchange with Google Artifact Registry.
  • Now (this MR):
    • Generate a JWT and get a glgo token.
    • Exchange that token with Google APIs to get a proper access token.
    • Use that access token to build the API client using the official Artifact Registry gem.

The "Now" version of the authentication flow seems more complex but there is a shortcut available to us. We can use a specific credentials file when building the API client that will take care of the 3 steps for us.

In other words, the "Now" version only requires: building the credentials structure and pass it to the official client. Done. 🎉

These changes above have been documented in issue GAR Integration: Custom client class (#425147 - closed).

Lastly, this is a saas only feature. As such, the client use is now gated behind this saas only feature. This change will have an impact on the location of the classes: they will now be moved to the EE side. This is in line with the expectations of the Artifact Registry integration to be available to gitlab.com only.

What does this MR do and why?

• Refactor the GCP artifact registry client class and base client class to implement:
  • the new authentication flow.
  • the additional API endpoints to support.
• Update the related specs.

This is part of a greater effort of integrating the Google Artifact Registry and that effort is behind a feature flag. As such, no changelog was provided here.

🚥 MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

✅

🖼 Screenshots or screen recordings

It's an API class so we don't really have an UI. To see the class in action, see the next action.

⚙ How to set up and validate locally

Although the client class itself is quite simple to use, the setup of the supporting components can be quite involved.

We have two way to set up things: either using the gcp demo project (if you have access) or using a service account.

1️⃣ Set up using the gcp demo project

⚠ This is only available to members with access to the gcp demo project.

1. In Cloud Run, there is a glgo instance running. Click on the details and copy the url.
2. In lib/google_cloud_platform/base_client.rb, replace the GLGO_BASE_URL constant with the url from (1.).
3. In lib/google_cloud_platform/jwt.rb, return a fixed string of your choice for #issuer. I used http://10io.gdk.test:8000.
4. Start your local GDK, access /oauth/discovery/keys and paste the content on a Gitlab.com snippet. Copy the url of the raw form of the snippet.
5. In Cloud Run, create a new version to deploy and update the GLGO_KNOWN_ISSUERS env variable with the following string: ,<issuer string>;<url of the raw form of the snippet>

Don't forget to set up a Workload Identity Federation properly and get its url without the protocol.

Regarding the gcp parameters, use the demo repository that is in the Artifact Registry.

2️⃣ Set up using a service account

1. Create an Artifact Registry repository of type docker images.
2. Push several images to that repository.
3. Create a service account that has the Artifact Registry Reader role.
4. Create a json file credentials and download it.
5. In lib/google_cloud_platform/artifact_registry/client.rb, in the #gcp_client function. Replace the config.credentials = line and set it to the path to the credentials file.

3️⃣ The client class in action

One last setup, there is a guard to make sure that the client class is used in the saas instance only. In lib/google_cloud_platform/artifact_registry/client.rb, comment L41.

Now, that the set up is out of the way, let's play! 🕹

One note on the client instance. It is meant to be built and used right away. The different tokens in play here have quite short life times. If you build the client object and call one of the methods a few minutes later, you will probably get the following error GoogleCloudPlatform::ArtifactRegistry::Client::ApiError: 14:Getting metadata from plugin failed with error: #<RuntimeError:"Unable to retrieve Identity Pool subject token {\"error\":\"failed to parse and validate input token: \\\"exp\\\" not satisfied\"}\n">}. That's basically saying that the token expired.

That's ok, how this client class is meant to be used is really to have the client initialization and the function call in a quick succession.

In a rails console:

client = GoogleCloudPlatform::ArtifactRegistry::Client.new(
  project: Project.first, 
  user: User.first, 
  gcp_project_id: '<google project id>', 
  gcp_location: '<google location of the repository>', 
  gcp_repository: '<google artifact registry repository name>', 
  gcp_wlif: '<workload identity federation url WITHOUT the protocol. Eg. start with //iam.googleapis.com/projects>'
)

client.repository
=> <Google::Cloud::ArtifactRegistry::V1::Repository: docker_config: <Google::Cloud::ArtifactRegistry::V1::Repository::DockerRepositoryConfig: immutable_tags: false>, name: "<the full name of the repository>", format: :DOCKER, description: "", labels: {}, create_time: <Google::Protobuf::Timestamp: seconds: 1702647962, nanos: 371683000>, update_time: <Google::Protobuf::Timestamp: seconds: 1703089638, nanos: 889259000>, kms_key_name: "", mode: :STANDARD_REPOSITORY, cleanup_policies: {}, size_bytes: 1028232179, satisfies_pzs: false, cleanup_policy_dry_run: true>

client.docker_images
=> <Google::Cloud::ArtifactRegistry::V1::ListDockerImagesResponse: 
    docker_images: [<Google::Cloud::ArtifactRegistry::V1::DockerImage ...>, <Google::Cloud::ArtifactRegistry::V1::DockerImage ...>], 
    next_page_token: "<next_page_token>"
>

client.docker_image(name: '<docker_image_name>')
=> <Google::Cloud::ArtifactRegistry::V1::DockerImage: ...>

You can also try exceptions:

client = GoogleCloudPlatform::ArtifactRegistry::Client.new(
  project: Project.first, 
  user: User.first, 
  gcp_project_id: '<google project id>', 
  gcp_location: 'invalid location', 
  gcp_repository: '<google artifact registry repository name>', 
  gcp_wlif: '<workload identity federation url WITHOUT the protocol. Eg. start with //iam.googleapis.com/projects>'
)
GoogleCloudPlatform::ArtifactRegistry::Client::ApiError: 3:Request contains an invalid argument.. debug_error_string:{...}