Skip to content

Add GCP Secret Manager secrets integration

Albert requested to merge alberts-gcp-secrets-manager into main

What does this MR do?

Add GCP Secret Manager secrets integration - gitlab#428400 (closed).

Why was this MR needed?

This is the Runner implementation for the epic to add GCP Secret Manager native integration, alongside the existing Vault and Azure KV integrations.

With this feature we will be adding a new CI configuration as such:

job_name:
  id_tokens:
    GCP_SM_ID_TOKEN:
      aud: gcp-workload-identity-federation-audience
  secrets:
    DATABASE_PASSWORD:
      gcp_secret_manager:
        name: my-project-secret  # This is the name of the secret defined in GCP Secrets Manager
        version: 1               # optional: default to `latest`.
      token: GCP_SM_ID_TOKEN

With this configuration, the job would have the secret retrieved from GCP Secret Manager stored in the build variable DATABASE_PASSWORD.

The implementation of the CI config in Rails is in this MR: gitlab!139201 (merged)

Implementation detail

Given the CI config above, Runner would need to perform the following steps to obtain the secret from GCP Secret Manager.

  1. Perform an authentication with GCP Secure Token Service (STS) using the id token and exchange it for a bearer token.
  2. Use the bearer token to authenticate to GCP Secret Manager and retrieve the secret version payload.
  3. Inject the secret into the build variables.

This MR adds the GCP resolver implementation for (1) and (2). Injecting the secret is already implemented through the secrets resolver process.

What's the best way to test this MR?

Video recording demoing the workflow from setting a gcp_secret_manager keyword to pulling it in the CI job: https://youtu.be/Utaxgu-_vis (internal).

It is a rather involved process to test this change. You'd need the following:

  1. A copy of the Rails implementation gitlab!139201 (merged)
  2. A GDK that is publicly accessible on HTTPS. This is required by GCP for OIDC verification. The GitLab git repo needs to be checked out to the branch of the MR gitlab!139201 (merged).
  3. Configure GCP Workload Identity Federation (see the video above for the detail).
  4. Create secrets in GCP Secret Manager.
  5. Configure GCP IAM to allow the Workload Identity Federation Principal to access the GCP Secret Manager secrets (see the vide above for the detail).
  6. Clone this branch and build the binary.
  7. Create a test project with the new CI config added in gitlab!139201 (merged).
  8. Add the required GCP related project variables. The list can be found in the Rails MR gitlab!139201 (merged).
  9. Register the runner and start it.
  10. Run the pipeline.

What are the relevant issue numbers?

Feature epic: &11739 (closed) Blueprint: https://docs.gitlab.com/ee/architecture/blueprints/ci_gcp_secrets_manager/ Rails implementation to add the CI config for gcp_secret_manager: gitlab#428401 (closed) Runner implementation (this MR): gitlab#428400 (closed)

Edited by Albert

Merge request reports