## Summary After secrets are fetched from the HashiCorp Vault vault integration, gitlab-runner performs variable expansion on the secret value like any other variable. Any passwords stored in vault that contain the '$' character are not cleanly passed through from the vault integration to the job. ## Feature Proposal - [ ] Mark variables fetched from secrets as `raw` - [ ] If a user relies on variable expansion, then the user can use `expand:true` ``` variables: RAW_VAR: value: $myvar expand: true ``` ## Steps to reproduce 1. Configure external secrets integration with HashiCorp Vault as per https://docs.gitlab.com/ee/ci/secrets/index.html 2. Add a secret in vault where the value contains a '$' sign 3. Retrieve the secret from vault in .gitlab.yml 4. The value of the secret available to the job is different from the value returned by the vault API <details> <summary> vault secret </summary> ```shell $ vault write secret/path/mysecret password='nota$ecurepassword' $ vault read secret/path/mysecret Key Value --- ----- refresh_interval 1h password nota$ecurepassword ``` </details> <details> <summary> .gitlab-ci.yml </summary> ```yml variables: VAULT_SERVER_URL: https://vault.example.com VAULT_AUTH_PATH: gitlab VAULT_AUTH_ROLE: gitlab-runner example-job: stage: build secrets: EXAMPLE_SECRET_PASSWORD: vault: engine: { name: kv-v1, path: secret } path: path/mysecret field: password image: name: busybox script: - fgrep '$' "${EXAMPLE_SECRET_PASSWORD}" || echo 'Dollar sign not in secret' - cat "${EXAMPLE_SECRET_PASSWORD}" ``` </details> ## Actual behavior script step output is: ``` $ fgrep '$' "${EXAMPLE_SECRET_PASSWORD}" || echo 'Dollar sign not in secret' Dollar sign not in secret $ cat "${EXAMPLE_SECRET_PASSWORD}" nota ``` where gitlab-runner seems to have attempted to expand the '$ecurepassword' part of the secret value as a variable ## Expected behavior ``` $ fgrep '$' "${EXAMPLE_SECRET_PASSWORD}" || echo 'Dollar sign not in secret' nota$ecurepassword $ cat "${EXAMPLE_SECRET_PASSWORD}" nota$ecurepassword ``` where the secret is unchanged from what was stored vault ## Relevant logs and/or screenshots <!-- Paste the job logs inside of the code blocks (```) below so it would be easier to read. --> <details> <summary> job log </summary> ```sh Running with gitlab-runner 13.11.0 (7f7a4bb0) on gitlab-runner-6fcbd6ddf4-8m85b HA9gXzNi Resolving secrets 00:02 Resolving secret "EXAMPLE_SECRET_PASSWORD"... Using "vault" secret resolver... Preparing the "kubernetes" executor 00:00 Using Kubernetes namespace: gitlab-runner Using Kubernetes executor with image busybox ... Preparing environment 00:04 Waiting for pod gitlab-runner/runner-ha9gxzni-project-13545-concurrent-0rc4w2 to be running, status is Pending Running on runner-ha9gxzni-project-13545-concurrent-0rc4w2 via gitlab-runner-6fcbd6ddf4-8m85b... Getting source from Git repository 00:00 Fetching changes with git depth set to 50... Initialized empty Git repository in /builds/mygroup/myproject/.git/ Created fresh repository. Checking out e1e215e7 as testcase... Skipping Git submodules setup Executing "step_script" stage of the job script 00:00 $ fgrep '$' "${EXAMPLE_SECRET_PASSWORD}" || echo 'Dollar sign not in secret' Dollar sign not in secret $ cat "${EXAMPLE_SECRET_PASSWORD}" nota Cleaning up file based variables 00:00 Job succeeded ``` </details> ## Environment description * gitlab-runner 13.11.0 * kubernetes executor - private runners in our own k8s cluster * GitLab Enterprise Edition 13.8.6-ee * vault kv_v1 secrets engine <!-- Are you using shared Runners on GitLab.com? Or is it a custom installation? Which executors are used? Please also provide the versions of related tools like `docker info` if you are using the Docker executor. --> <!-- Please add the contents of `config.toml` inside of the code blocks (```) below, remember to remove any secret tokens! --> <details> <summary> config.toml contents </summary> ```toml listen_address = ":9252" concurrent = 10 check_interval = 30 log_level = "info" [session_server] session_timeout = 1800 [[runners]] name = "gitlab-runner-6fcbd6ddf4-8m85b" request_concurrency = 1 url = "https://gitlab.example.com/" token = "redacted" executor = "kubernetes" [runners.custom_build_dir] [runners.cache] [runners.cache.s3] [runners.cache.gcs] [runners.cache.azure] [runners.kubernetes] host = "" bearer_token_overwrite_allowed = false image = "ubuntu:16.04" namespace = "gitlab-runner" namespace_overwrite_allowed = "" privileged = false pull_policy = [""] helper_image = "gitlab/gitlab-runner-helper:x86_64-v${CI_RUNNER_VERSION}" service_account_overwrite_allowed = "" pod_annotations_overwrite_allowed = "" [runners.kubernetes.affinity] [runners.kubernetes.pod_security_context] [runners.kubernetes.volumes] [runners.kubernetes.dns_config] ``` </details> ### Used GitLab Runner version <!-- Please run and paste the output of `gitlab-runner --version`. If you are using a Runner where you don't have access to, please paste at least the first lines the from build log, like: ``` Running with gitlab-ci-multi-runner 1.4.2 (bcc1794) Using Docker executor with image golang:1.8 ... ``` --> ``` Version: 13.11.0 Git revision: 7f7a4bb0 Git branch: 13-11-stable GO version: go1.13.8 Built: 2021-04-20T17:02:30+0000 OS/Arch: linux/amd64 ``` ## Possible fixes The problem does not seem specific to vault at all, but the way secrets are expanded as any other variable by `GetAllVariables()` in build.go : https://gitlab.com/gitlab-org/gitlab-runner/blob/0b1fbf16e10e5f38ecdeff8f8b07f69a34489b0e/common/build.go#L1141 This should be solvable by delaying appending `b.secretsVariables` to `variables` until after the `variables.Expand()` call here: https://gitlab.com/gitlab-org/gitlab-runner/blob/0b1fbf16e10e5f38ecdeff8f8b07f69a34489b0e/common/build.go#L1143 This possible fix would introduce a regression (and break the workaround below) where secrets would no longer be able to be expanded when referenced inside other variables. A fix that would avoid that regression would be to just overwrite the values of any secrets in `variables` with the values in `b.secretsVariables` immediately after expansion. ## ~~Workaround~~ _Update: there is no workaround. See comment_ ~~As secrets are treated as any other File variable after resolution, and that gitlab-runner does not recursively expand variables, the original secret can be accessed by wrapping it in a (hopefully masked) variable. e.g.:~~ <details> <summary> .gitlab-ci.yml </summary> ```yml variables: VAULT_SERVER_URL: https://vault.example.com VAULT_AUTH_PATH: gitlab VAULT_AUTH_ROLE: gitlab-runner example-job: stage: build secrets: EXAMPLE_SECRET_PASSWORD_REF: vault: engine: { name: kv-v1, path: secret } path: path/mysecret field: password variables: EXAMPLE_SECRET_PASSWORD: "$EXAMPLE_SECRET_PASSWORD_REF" image: name: busybox script: - fgrep '$' "${EXAMPLE_SECRET_PASSWORD}" || echo 'Dollar sign not in secret' - cat "${EXAMPLE_SECRET_PASSWORD}" ``` </details>