Inject dynamic environment variables

Description

Problem to Solve

There is currently no way to inject dynamic environment variables into a build. This can be accomplished by adding a call to a user-provided external service when defining the build environment variables.

Use Cases

The user-provided external HTTP service could fetch ephemeral credentials, e.g. make a call to a Hashicorp Vault instance to retrieve a token with an expiry date, which could then be used to in the build to fetch additional secrets from Vault.

Proposal

Add runner configuration for a user-provided external HTTP service that can be defined as flags or environment variables to the register command or as attributes in the config.toml.

Example config.toml:

[[runners]]
  ...

  [runners.external_variables]
    # Target system to talk to, can be a unix socket like unix:///var/run/docker.sock
    url = "https://secure.example.com/my-service"
    # When set it will use ca.pem, cert.pem and key.pem from that folder to make secure TLS connection to Docker (useful in boot2docker)
    tls_cert_path = "/secure/tls"

Example register commands:

# As flags
$ gitlab-runner register --external-variables-url="https://secure.example.com/my-service" --external-variables-cert-path="/secure/tls" ...

# As environment variables
$ export EXTERNAL_VARIABLES_URL=https://secure.example.com/my-service
$ export EXTERNAL_VARIABLES_CERT_PATH=/secure/tls
$ gitlab-runner register

Specifically, we could call a new function in GetAllVariables() that would take in all prior environment variables as a parameter, like GetExternalVariables(variables JobVariables), send a POST to the external service with these variables as the JSON body, and include the response body to the runner as additional environment variables in the build.

     ┌──────┐                        ┌───────┐                       
     │Runner│                        │Service│                       
     └──┬───┘                        └───┬───┘                       
        │    POST build variables        │                           
        │    as JSON to provided URI     │                           
        │ ───────────────────────────────>                           
        │                                │                           
        │                                │────┐                      
        │                                │    │ Make additional calls
        │                                │<───┘ & logic as needed    
        │                                │                           
        │                                │                           
        │ Additional variables as JSON   │                           
        │ <─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ -                           
        │                                │                           
        │                                │
        │────┐                           │                         
        │    │ Add environment variables │                         
        │<───┘ into build                │                          
        │                                │                            
     ┌──┴───┐                        ┌───┴───┐                       
     │Runner│                        │Service│                       
     └──────┘                        └───────┘      

The external service is user-provided as it will be different based on the needs of the teams using this feature, and should follow the following contract with the runner.

For a 200 response from the service with a JSON payload like the following, the runner adds CUSTOM_VAR_1 and CUSTOM_VAR_2 into the build's environment variables.

{
  "CUSTOM_VAR_1": "1",
  "CUSTOM_VAR_2": "2",
}

For a non-200 response from the service with a JSON payload with an error key, the runner will bubble up the error value as a warning in the build log to the user.

Response JSON body:

{
  "error": "some error message"
}

Any failures from the service will fail the build at start and display the bubbled-up error in the build log:

ERROR: Preparation of dynamic environment variables failed: some error message

Links to related issues and merge requests / references

The proposal comes from spiking on https://gitlab.com/gitlab-org/gitlab-ce/issues/52510.