Skip to content

Add CI config for GCP Secret Manager

Albert requested to merge 428401-gcp-sm-ci-secrets into master

What does this MR do and why?

Add CI config for GCP Secret Manager.

The secret manager is added as gcp_secret_manager keyword on secrets. An example of a yaml would be as follows:

job_name:
  id_tokens:
    GCP_SM_ID_TOKEN:
      aud: https://iam.googleapis.com/projects/$GCP_PROJECT_NUMBER/locations/global/workloadIdentityPools/$GCP_WORKLOAD_FEDERATION_POOL_ID/providers/$GCP_WORKLOAD_FEDERATION_PROVIDER_ID # or a custom audience as configured in GCP Workload Identity Pool Provider.
  secrets:
    DATABASE_PASSWORD:
      gcp_secret_manager:
        name: my-project-secret  # This is the name of the secret defined in GCP Secret Manager
        version: '1'               # optional: default to `latest`.
      token: GCP_SM_ID_TOKEN

In addition to the above config, the user needs to provide the following variables:

  • GCP Project Number GCP_PROJECT_NUMBER
  • GCP Workload Identity Federation Pool ID GCP_WORKLOAD_IDENTITY_FEDERATION_POOL_ID
  • GCP Workload Identity Federation Provider ID GCP_WORKLOAD_IDENTITY_FEDERATION_PROVIDER_ID

Given the above yaml and CI variables, the job payload to Runner should contain the following json:

{
  "secrets": {
    "DATABASE_PASSWORD": {
      "gcp_secret_manager": {
        "name": "my-project-secret",
        "version": "1",
        "server": {
          "project_number": "1234",
          "workload_identity_federation_pool_id": "pool-id",
          "workload_identity_federation_provider_id": "provider-id",
          "jwt": "$GCP_SM_ID_TOKEN"
        }
      }
    }
  }
}

The runner implementation to handle this payload is done in gitlab-runner!4512 (merged).

Screenshots or screen recordings

The recording can be found in this GitLab Unfiltered video (internal)

Before After

How to set up and validate locally

It is a rather involved process to test this change. You'd need the following:

  1. A copy of the Runner implementation gitlab-runner!4512 (merged)
  2. A GDK that is publicly accessible on HTTPS. This is required by GCP for OIDC verification. The GitLab git repo needs to be checked out to the branch of the MR !139201 (merged). For security reason, it is recommended to run the GDK on a VM and expose the VM, instead of your laptops.
  3. Configure GCP Workload Identity Federation (see the video above for the detail).
  4. Create secrets in GCP Secret Manager.
  5. Configure GCP IAM to allow the Workload Identity Federation Principal to access the GCP Secret Manager secrets (see the vide above for the detail).
  6. Pull the Runner development branch and build the runner binary.
  7. Create a test project with the new CI config added in !139201 (merged).
  8. Add the required GCP related project variables.
  9. Register the runner and start it.
  10. Run the pipeline.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #428401 (closed)

Edited by Albert

Merge request reports