Skip to content

Support Google Cloud Secrets from projects which are not the same as the WIF pool

Summary

When a Google Workload Identity Federation pool exists in Project A and the secret one wishes to access is in Project B, there is no way of configuring secrets:...:gcp_secret_manager to express this.

Google recommends using a single WIF pool per provider and so we have configured a single WIF pool representing our on-prem "GitLab" in a shared Google project.

We are heavy Google Cloud users and so our various products sit within their own unrelated Google Cloud projects. Each of these products have their own secrets which we would like to expose to GitLab CI runners. At our scale, the secrets cannot, in general, live in the same project as the WIF pool and neither, AIUI, do they need to since GitLab can exchange the GitLab minted JWT token for a Google WIF access token and the IAM principal corresponding to that WIF token can appear in any Cloud IAM policy in any project.

Steps to reproduce

  1. Create a WIF Pool in Google Project A following https://docs.gitlab.com/ee/ci/secrets/gcp_secret_manager.html
  2. Create a Google Secret in Project B.
  3. Grant the appropriate principalSet://... IAM principal access to the secret via an IAM policy.
  4. Attempt to configure Google Cloud Secret Manager integration as per https://docs.gitlab.com/ee/ci/secrets/gcp_secret_manager.html
.gitlab-ci.yml 
# Project A is redacted as project 1234565789. Project B is redacted as 987654321
#
# Secret is located at projects/987654321/secrets/my-secret/versions/1
build:
  id_tokens:
    GCP_ID_TOKEN:
      aud: //iam.googleapis.com/projects/1234565789/locations/global/workloadIdentityPools/gitlab/providers/gitlab
  secrets:
    SUPER_SECRET:
      gcp_secret_manager:
        name: my-secret
        version: 1
      token: $GCP_ID_TOKEN
  variables:
    GCP_PROJECT_NUMBER: "1234565789"
    GCP_WORKLOAD_IDENTITY_FEDERATION_POOL_ID: gitlab
    GCP_WORKLOAD_IDENTITY_FEDERATION_PROVIDER_ID: gitlab
  script:
    - echo $SUPER_SECRET

Actual behavior

Project A is represented as project 1234565789. Project B is represented as 987654321. Secret is located at projects/987654321/secrets/my-secret/versions/1.

CI job log (redacted). Note the incorrect project id being used to fetch the secret:

Running with gitlab-runner 16.8.1 (a6097117)
  on REDACTED, system ID: REDACTED
Resolving secrets 00:00
Resolving secret "SUPER_SECRET"...
Using "gcp_secret_manager" secret resolver...
ERROR: Job failed (system failure): resolving secrets: failed to get secret: rpc error: code = PermissionDenied desc = Permission 'secretmanager.versions.access' denied for resource 'projects/1234565789/secrets/my-secret/versions/1' (or it may not exist).

If gcp_secret_manager:name: is set to projects/987654321/secrets/my-secret/versions/1 instead, the failure message is:

ERROR: Job failed (system failure): resolving secrets: failed to get secret: rpc error: code = InvalidArgument desc = The provided Secret ID [projects/1234565789/secrets/projects/987654321/secrets/my-secret/versions/1] does not match the expected format [projects/*/secrets/*/versions/*]

If GCP_PROJECT_NUMBER is set to the project containing the secret instead, the failure message is:

ERROR: Job failed (system failure): resolving secrets: failed to exchange sts token: googleapi: got HTTP response code 400 with body: {"error":"invalid_target","error_description":"The target service indicated by the \"audience\" parameters is invalid. This might either be because the pool or provider is disabled or deleted or because it doesn't exist."}

Expected behavior

Running with gitlab-runner 16.8.1 (a6097117)
  on REDACTED, system ID: REDACTED
Resolving secrets 00:00
Resolving secret "SUPER_SECRET"...
Using "gcp_secret_manager" secret resolver...
...
{CONTENTS OF SECRET}

Relevant logs and/or screenshots

job log 
Running with gitlab-runner 16.8.1 (a6097117)
  on REDACTED, system ID: REDACTED
Resolving secrets 00:00
Resolving secret "SUPER_SECRET"...
Using "gcp_secret_manager" secret resolver...
ERROR: Job failed (system failure): resolving secrets: failed to get secret: rpc error: code = PermissionDenied desc = Permission 'secretmanager.versions.access' denied for resource 'projects/1234565789/secrets/my-secret/versions/1' (or it may not exist).

Environment description

We are using GitLab runners started in a k8s cluster via the helm chart. The same behaviour is observed using GitLab runner running on bare VMs.

config.toml contents 
# Available on request

Used GitLab Runner version

Running with gitlab-runner 16.8.1 (a6097117)

Possible fixes

The main issue is in the following lines:

The same project id is used to construct both the WIF pool audience (in stsAudience) and the Google Secret resource name (in secretVersionResourceName).

Potential mitigations:

  • Add an optional GCP_SECRET_PROJECT_NUMBER variable which is used to construct the Google Cloud secret resource name
  • (My preference) allow specifying the full projects/{project-id}/secrets/{secret-id} resource name in secrets:...:gcp_secret_manager. This requires no additional configuration, preserves existing behaviour and provides a way to explicitly specify the Google secret's project if necessary. Code changes would be limited to the secretVersionResourceName function.
Edited by Rich Wareham