Introduce an explicit concept of `secrets:` in GitLab CI
Problem to solve
TL;DR The usage of variables:
is confusing today and we want to add a proper secrets handling.
We had awesome discussion with @grzesiek and @krasio about Vault integration and Secrets.
This extends the proposal from !31831 (closed) and #28321 (closed).
Proposal
secrets:
We decided that we see a big value in having explicit secrets:
- The
secrets:
are different thanvariables:
- The
secrets:
clearly indicate a purpose: We want a secret to be used! - The
variables:
can be used by variable expansion and are by definition "known" - The
secrets:
can be only used during the execution of the CI job by the Runner - We have many credential providers that can be used by
secrets:
- The CI job if it wants to use any secret needs to request it explicitly, only requested are exposed
- Secrets create a clear split between
variables:
andsecrets:
on their purpose - Secrets access should be auditable by know: who, when and for what
- All secrets are masked, always
Vault integration
- Vault is one of the secret providers
- We want Vault to be easy to be use in default case, but allow flexibility for our customers that have different approach
- We want Vault auth in general to be configured via UI, but not for the first iteration!
- Today, we consider Vault to be configured using CI Variables:
VAULT_URL
,VAULT_AUTH_...
. This allows us to have a single vault per job, but many different vaults used by different jobs, as theseVAULT_*
variables can be defined for a specific environment scope or in context of a specific job - We offer a simple opinionated syntax, we use by default
kv2
engine - We convert a simple syntax (a string) into a complex hash having all details explicitly stated. This is how we process secrets internally, but we expose an easy to start usage of Vault.
- We allow users to fine-configure all details if the simple syntax is not enough for them
Vault is configured by adding to CI Variables (can be via CI/CD Settings, or part of YAML):
-
VAULT_URL
: this is required -
VAULT_AUTH_*
(username, password, role): this is optional
secrets:
# production/s3 is a `path:`, where `secret-key` is a fieldname
# @path/to/engine indicates a path used by `kv-v2` which we consider to be a default engine
VAULT_SIMPLE_SECRET:
vault: production/s3/secret-key
file: true/false # optional, with default: false
# false: VAULT_SIMPLE_SECRET would hold a masked value
# true: VAULT_SIMPLE_SECRET would hold a path to ephemeral file holding a secret
VAULT_SIMPLE2_SECRET:
vault: production/s3/secret-key@path/to/engine
VAULT_COMPLEX_SECRET:
vault:
engine:
name: kv-v2
path: path/to/engine
path: production/s3
field: secret-key
Rethinking the purpose of CI/CD Variables (mid-term future?)
- CI/CD Variables are multipurpose and confusing as they are a "known values", like
MY_URL
and "secrets", likeAWS_S3_SECRET_KEY
- All variables are always exposed to each CI job which has a security consequences, as it increases a chance of exposing secret when it does not need it
- GitLab can be another secret provider (aka.
internal
), similar to other secret providers that we implement - GitLab does offer an internal Secret Provider
- One of the ideas behind internal secret provider is to mark some Variables as of type
Secret
- Another idea is to split the current variables into being
Variables
andSecrets
- Secret as described earlier needs to be written explicitly in CI job as required to be used
We would
secrets:
MY_INTERNAL_SECRET:
gitlab:
file: true/false # optional, with default: false
# false: MY_INTERNAL_SECRET would hold a masked value
# true: MY_INTERNAL_SECRET would hold a path to ephemeral file holding a secret
Other credentials storage (future)
Any other credential storage takes a path of vault:
following the same design pattern, like:
secrets:
MY_CREDHUB_SECRET:
credhub: path/to/secret
Intended users
Links / references
Edited by Krasimir Angelov