Mark secrets values fetched from Vault integration as raw
Summary
After secrets are fetched from the HashiCorp Vault vault integration, gitlab-runner performs variable expansion on the secret value like any other variable.
Any passwords stored in vault that contain the '$' character are not cleanly passed through from the vault integration to the job.
Feature Proposal
-
Mark variables fetched from secrets as raw -
If a user relies on variable expansion, then the user can use expand:true
variables:
RAW_VAR:
value: $myvar
expand: true
Steps to reproduce
- Configure external secrets integration with HashiCorp Vault as per https://docs.gitlab.com/ee/ci/secrets/index.html
- Add a secret in vault where the value contains a '$' sign
- Retrieve the secret from vault in .gitlab.yml
- The value of the secret available to the job is different from the value returned by the vault API
vault secret
$ vault write secret/path/mysecret password='nota$ecurepassword'
$ vault read secret/path/mysecret
Key Value
--- -----
refresh_interval 1h
password nota$ecurepassword.gitlab-ci.yml
variables:
VAULT_SERVER_URL: https://vault.example.com
VAULT_AUTH_PATH: gitlab
VAULT_AUTH_ROLE: gitlab-runner
example-job:
stage: build
secrets:
EXAMPLE_SECRET_PASSWORD:
vault:
engine: { name: kv-v1, path: secret }
path: path/mysecret
field: password
image:
name: busybox
script:
- fgrep '$' "${EXAMPLE_SECRET_PASSWORD}" || echo 'Dollar sign not in secret'
- cat "${EXAMPLE_SECRET_PASSWORD}"Actual behavior
script step output is:
$ fgrep '$' "${EXAMPLE_SECRET_PASSWORD}" || echo 'Dollar sign not in secret'
Dollar sign not in secret
$ cat "${EXAMPLE_SECRET_PASSWORD}"
notawhere gitlab-runner seems to have attempted to expand the '$ecurepassword' part of the secret value as a variable
Expected behavior
$ fgrep '$' "${EXAMPLE_SECRET_PASSWORD}" || echo 'Dollar sign not in secret'
nota$ecurepassword
$ cat "${EXAMPLE_SECRET_PASSWORD}"
nota$ecurepasswordwhere the secret is unchanged from what was stored vault
Relevant logs and/or screenshots
job log
Running with gitlab-runner 13.11.0 (7f7a4bb0)
on gitlab-runner-6fcbd6ddf4-8m85b HA9gXzNi
Resolving secrets
00:02
Resolving secret "EXAMPLE_SECRET_PASSWORD"...
Using "vault" secret resolver...
Preparing the "kubernetes" executor
00:00
Using Kubernetes namespace: gitlab-runner
Using Kubernetes executor with image busybox ...
Preparing environment
00:04
Waiting for pod gitlab-runner/runner-ha9gxzni-project-13545-concurrent-0rc4w2 to be running, status is Pending
Running on runner-ha9gxzni-project-13545-concurrent-0rc4w2 via gitlab-runner-6fcbd6ddf4-8m85b...
Getting source from Git repository
00:00
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/mygroup/myproject/.git/
Created fresh repository.
Checking out e1e215e7 as testcase...
Skipping Git submodules setup
Executing "step_script" stage of the job script
00:00
$ fgrep '$' "${EXAMPLE_SECRET_PASSWORD}" || echo 'Dollar sign not in secret'
Dollar sign not in secret
$ cat "${EXAMPLE_SECRET_PASSWORD}"
nota
Cleaning up file based variables
00:00
Job succeededEnvironment description
- gitlab-runner 13.11.0
- kubernetes executor - private runners in our own k8s cluster
- GitLab Enterprise Edition 13.8.6-ee
- vault kv_v1 secrets engine
config.toml contents
listen_address = ":9252"
concurrent = 10
check_interval = 30
log_level = "info"
[session_server]
session_timeout = 1800
[[runners]]
name = "gitlab-runner-6fcbd6ddf4-8m85b"
request_concurrency = 1
url = "https://gitlab.example.com/"
token = "redacted"
executor = "kubernetes"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.kubernetes]
host = ""
bearer_token_overwrite_allowed = false
image = "ubuntu:16.04"
namespace = "gitlab-runner"
namespace_overwrite_allowed = ""
privileged = false
pull_policy = [""]
helper_image = "gitlab/gitlab-runner-helper:x86_64-v${CI_RUNNER_VERSION}"
service_account_overwrite_allowed = ""
pod_annotations_overwrite_allowed = ""
[runners.kubernetes.affinity]
[runners.kubernetes.pod_security_context]
[runners.kubernetes.volumes]
[runners.kubernetes.dns_config]Used GitLab Runner version
Version: 13.11.0
Git revision: 7f7a4bb0
Git branch: 13-11-stable
GO version: go1.13.8
Built: 2021-04-20T17:02:30+0000
OS/Arch: linux/amd64Possible fixes
The problem does not seem specific to vault at all, but the way secrets are expanded as any other variable by GetAllVariables() in build.go :
This should be solvable by delaying appending b.secretsVariables to variables until after the variables.Expand() call here:
This possible fix would introduce a regression (and break the workaround below) where secrets would no longer be able to be expanded when referenced inside other variables.
A fix that would avoid that regression would be to just overwrite the values of any secrets in variables with the values in b.secretsVariables immediately after expansion.
Workaround
Update: there is no workaround. See comment
As secrets are treated as any other File variable after resolution, and that gitlab-runner does not recursively expand variables, the original secret can be accessed by wrapping it in a (hopefully masked) variable. e.g.:
.gitlab-ci.yml
variables:
VAULT_SERVER_URL: https://vault.example.com
VAULT_AUTH_PATH: gitlab
VAULT_AUTH_ROLE: gitlab-runner
example-job:
stage: build
secrets:
EXAMPLE_SECRET_PASSWORD_REF:
vault:
engine: { name: kv-v1, path: secret }
path: path/mysecret
field: password
variables:
EXAMPLE_SECRET_PASSWORD: "$EXAMPLE_SECRET_PASSWORD_REF"
image:
name: busybox
script:
- fgrep '$' "${EXAMPLE_SECRET_PASSWORD}" || echo 'Dollar sign not in secret'
- cat "${EXAMPLE_SECRET_PASSWORD}"