GitLab reads from Vault for CI Variables
Problem to solve
Secrets stored in HashiCorp Vault are currently not easily read by GitLab for CI Jobs. We want to support a user that has secret stored in Vault and have it used by GitLab.
This feature will be used by automation developers who need to access secrets as part of their build and deploy process, providing a new and more secure way to do so.
We are supporting the leanest approach for GitLab read a secret from Vault and bring that into a CI Job, using the JWT as an authentication method. The secret would be stored in Vault and retrieved from Vault via the yaml.
We'll add support for the following syntax to
.gitlab-ci.yml in order to define Vault servers and secrets from the syntax in #218746 (closed). We will also be delivering the Runner configuration as described in #212252.
secrets: # production/s3 is a `path:`, where `secret-key` is a fieldname # @path/to/engine indicates a path used by `kv-v2` which we consider to be a default engine VAULT_SIMPLE_SECRET: vault: production/s3/secret-key VAULT_SIMPLE2_SECRET: vault: production/s3/secret-key@path/to/engine VAULT_COMPLEX_SECRET: vault: engine: name: kv-v2 path: path/to/engine path: production/s3 field: secret-key
GitLab will parse this section and provide the data to the Runner as part of the job request response. Runner will then authenticate with Vault, fetch the secrets and save them under a path that will be stored in the specified ENV variable (similar to how "File" type CI variables work).
In the first iteration the Vault server definition (url and auth method path) will be defined with CI variables, later we'll add UI for this with #218677.
- Start with YAML --> in the future move this into a group or project settings like the Variables framework - will always need to have a section in the YAML
- Once we have the section in the job YAML, we should continue to expand the YAML
Permissions and Security
Permissions are configured at Vault by using policies, roles, and when configuring the auth method. This should be documented with example, similar to https://docs.gitlab.com/ee/ci/examples/authenticating-with-hashicorp-vault.
- This will need documentation on how this works
- Most of the secret management will be contained on the Vault side. For example, secrets expiration and storage will be handled on the Vault Side
_ We'll need vault instance accessible over the internet in order to test this