Inject dynamic environment variables
Description
Problem to Solve
There is currently no way to inject dynamic environment variables into a build. This can be accomplished by adding a call to a user-provided external service when defining the build environment variables.
Use Cases
The user-provided external HTTP service could fetch ephemeral credentials, e.g. make a call to a Hashicorp Vault instance to retrieve a token with an expiry date, which could then be used to in the build to fetch additional secrets from Vault.
Proposal
Add runner configuration for a user-provided external HTTP service that can be defined as flags or environment variables to the register
command or as attributes in the config.toml
.
Example config.toml
:
[[runners]]
...
[runners.external_variables]
# Target system to talk to, can be a unix socket like unix:///var/run/docker.sock
url = "https://secure.example.com/my-service"
# When set it will use ca.pem, cert.pem and key.pem from that folder to make secure TLS connection to Docker (useful in boot2docker)
tls_cert_path = "/secure/tls"
Example register
commands:
# As flags
$ gitlab-runner register --external-variables-url="https://secure.example.com/my-service" --external-variables-cert-path="/secure/tls" ...
# As environment variables
$ export EXTERNAL_VARIABLES_URL=https://secure.example.com/my-service
$ export EXTERNAL_VARIABLES_CERT_PATH=/secure/tls
$ gitlab-runner register
Specifically, we could call a new function in GetAllVariables() that would take in all prior environment variables as a parameter, like GetExternalVariables(variables JobVariables)
, send a POST to the external service with these variables as the JSON body, and include the response body to the runner as additional environment variables in the build.
┌──────┐ ┌───────┐
│Runner│ │Service│
└──┬───┘ └───┬───┘
│ POST build variables │
│ as JSON to provided URI │
│ ───────────────────────────────>
│ │
│ │────┐
│ │ │ Make additional calls
│ │<───┘ & logic as needed
│ │
│ │
│ Additional variables as JSON │
│ <─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ -
│ │
│ │
│────┐ │
│ │ Add environment variables │
│<───┘ into build │
│ │
┌──┴───┐ ┌───┴───┐
│Runner│ │Service│
└──────┘ └───────┘
The external service is user-provided as it will be different based on the needs of the teams using this feature, and should follow the following contract with the runner.
For a 200 response from the service with a JSON payload like the following, the runner adds CUSTOM_VAR_1
and CUSTOM_VAR_2
into the build's environment variables.
{
"CUSTOM_VAR_1": "1",
"CUSTOM_VAR_2": "2",
}
For a non-200 response from the service with a JSON payload with an error
key, the runner will bubble up the error
value as a warning in the build log to the user.
Response JSON body:
{
"error": "some error message"
}
Any failures from the service will fail the build at start and display the bubbled-up error in the build log:
ERROR: Preparation of dynamic environment variables failed: some error message
Links to related issues and merge requests / references
The proposal comes from spiking on https://gitlab.com/gitlab-org/gitlab-ce/issues/52510.