## Summary When a Google Workload Identity Federation pool exists in Project A and the secret one wishes to access is in Project B, there is no way of configuring `secrets:...:gcp_secret_manager` to express this. Google recommends using a single WIF pool per provider and so we have configured a single WIF pool representing our on-prem "GitLab" in a shared Google project. We are heavy Google Cloud users and so our various products sit within their own unrelated Google Cloud projects. Each of these products have their own secrets which we would like to expose to GitLab CI runners. At our scale, the secrets cannot, in general, live in the same project as the WIF pool and neither, AIUI, do they need to since GitLab can exchange the GitLab minted JWT token for a Google WIF access token and the IAM principal corresponding to that WIF token can appear in any Cloud IAM policy in any project. ## Steps to reproduce 1. Create a WIF Pool in Google Project A following https://docs.gitlab.com/ee/ci/secrets/gcp_secret_manager.html 2. Create a Google Secret in Project B. 3. Grant the appropriate `principalSet://...` IAM principal access to the secret via an IAM policy. 4. Attempt to configure Google Cloud Secret Manager integration as per https://docs.gitlab.com/ee/ci/secrets/gcp_secret_manager.html <!-- What do you need to do to reproduce the bug? Please include job definitions or git repository structure if relevant --> <!-- Please add the definition of the job from `.gitlab-ci.yml` that is failing inside of the code blocks (```) below. --> <details> <summary> .gitlab-ci.yml </summary> ```yml # Project A is redacted as project 1234565789. Project B is redacted as 987654321 # # Secret is located at projects/987654321/secrets/my-secret/versions/1 build: id_tokens: GCP_ID_TOKEN: aud: //iam.googleapis.com/projects/1234565789/locations/global/workloadIdentityPools/gitlab/providers/gitlab secrets: SUPER_SECRET: gcp_secret_manager: name: my-secret version: 1 token: $GCP_ID_TOKEN variables: GCP_PROJECT_NUMBER: "1234565789" GCP_WORKLOAD_IDENTITY_FEDERATION_POOL_ID: gitlab GCP_WORKLOAD_IDENTITY_FEDERATION_PROVIDER_ID: gitlab script: - echo $SUPER_SECRET ``` </details> ## Actual behavior <!-- What actually happens --> Project A is represented as project 1234565789. Project B is represented as 987654321. Secret is located at projects/987654321/secrets/my-secret/versions/1. CI job log (redacted). Note the incorrect project id being used to fetch the secret: ``` Running with gitlab-runner 16.8.1 (a6097117) on REDACTED, system ID: REDACTED Resolving secrets 00:00 Resolving secret "SUPER_SECRET"... Using "gcp_secret_manager" secret resolver... ERROR: Job failed (system failure): resolving secrets: failed to get secret: rpc error: code = PermissionDenied desc = Permission 'secretmanager.versions.access' denied for resource 'projects/1234565789/secrets/my-secret/versions/1' (or it may not exist). ``` If `gcp_secret_manager:name:` is set to `projects/987654321/secrets/my-secret/versions/1` instead, the failure message is: ``` ERROR: Job failed (system failure): resolving secrets: failed to get secret: rpc error: code = InvalidArgument desc = The provided Secret ID [projects/1234565789/secrets/projects/987654321/secrets/my-secret/versions/1] does not match the expected format [projects/*/secrets/*/versions/*] ``` If `GCP_PROJECT_NUMBER` is set to the project containing the secret instead, the failure message is: ``` ERROR: Job failed (system failure): resolving secrets: failed to exchange sts token: googleapi: got HTTP response code 400 with body: {"error":"invalid_target","error_description":"The target service indicated by the \"audience\" parameters is invalid. This might either be because the pool or provider is disabled or deleted or because it doesn't exist."} ``` ## Expected behavior <!-- What you should see instead --> ``` Running with gitlab-runner 16.8.1 (a6097117) on REDACTED, system ID: REDACTED Resolving secrets 00:00 Resolving secret "SUPER_SECRET"... Using "gcp_secret_manager" secret resolver... ... {CONTENTS OF SECRET} ``` ## Relevant logs and/or screenshots <!-- Paste the job logs inside of the code blocks (```) below so it would be easier to read. --> <details> <summary> job log </summary> ```sh Running with gitlab-runner 16.8.1 (a6097117) on REDACTED, system ID: REDACTED Resolving secrets 00:00 Resolving secret "SUPER_SECRET"... Using "gcp_secret_manager" secret resolver... ERROR: Job failed (system failure): resolving secrets: failed to get secret: rpc error: code = PermissionDenied desc = Permission 'secretmanager.versions.access' denied for resource 'projects/1234565789/secrets/my-secret/versions/1' (or it may not exist). ``` </details> ## Environment description We are using GitLab runners started in a k8s cluster via the helm chart. The same behaviour is observed using GitLab runner running on bare VMs. <!-- Are you using shared Runners on GitLab.com? Or is it a custom installation? Which executors are used? Please also provide the versions of related tools like `docker info` if you are using the Docker executor. --> <!-- Please add the contents of `config.toml` inside of the code blocks (```) below, remember to remove any secret tokens! --> <details> <summary> config.toml contents </summary> ```toml # Available on request ``` </details> ### Used GitLab Runner version ``` Running with gitlab-runner 16.8.1 (a6097117) ``` ## Possible fixes The main issue is in the following lines: * https://gitlab.com/gitlab-org/gitlab-runner/-/blob/main/helpers/gcp_secret_manager/service/gcp_secret_manager.go#L84 * https://gitlab.com/gitlab-org/gitlab-runner/-/blob/main/helpers/gcp_secret_manager/service/gcp_secret_manager.go#L114 The same project id is used to construct both the WIF pool audience (in `stsAudience`) *and* the Google Secret resource name (in `secretVersionResourceName`). Potential mitigations: * Add an optional `GCP_SECRET_PROJECT_NUMBER` variable which is used to construct the Google Cloud secret resource name * (My preference) allow specifying the full `projects/{project-id}/secrets/{secret-id}` resource name in `secrets:...:gcp_secret_manager`. This requires no additional configuration, preserves existing behaviour and provides a way to explicitly specify the Google secret's project if necessary. Code changes would be limited to the `secretVersionResourceName` function.