Expand environment variables in machine options

What does this MR do?

Adds the FF_MACHINE_OPTIONS_EXPAND_ENV feature flag, off by default. When it is on, $VAR and ${VAR} in MachineOptions and MachineOptionsWithName are replaced from the runner process environment when a machine is created (token and url have had the same expansion at config load since !6068 (merged)). MachineOptionsWithName are formatted with the machine name first and expanded after, so %s is unaffected.

${GITLAB_RUNNER_MACHINE_NAME} is the one variable not read from the environment: it is the name of the machine being created, so MachineOptions can reference the machine without a %s template.

Why was this MR needed?

Some machine labels are only known at runtime. The runner_manager_name label on our GCE instances carries the cluster name today because that is all the Helm template knows. With expansion it can carry the pod name of the manager replica that created the machine (google-label=runner_manager_name:${HOSTNAME}), which is what that label meant on the old VM-based managers and what you need when tracing a stray machine back to a manager.

docker-machine can't do this on its own. The drivers read credentials from their own environment variables, but google-label is a repeatable flag mixed with static values from the config, so a per-label value has to come through the runner config.

This is the earlier MR reopened. It was closed when the credentials use case went to the drivers. The change is now behind a feature flag, and the example and tests are the label case.

Why a feature flag

A literal $ in an existing config would start expanding, usually to an empty string. Off by default, flip the default later with a deprecation note, then remove.

What's the best way to test this MR?

[[runners]]
  executor = "docker+machine"
  [runners.machine]
    MachineDriver = "google"
    MachineName = "runner-%s"
    MachineOptions = [
      "google-project=...",
      "google-label=runner_manager_name:${HOSTNAME}",
    ]
  [runners.feature_flags]
    FF_MACHINE_OPTIONS_EXPAND_ENV = true

Created instances get runner_manager_name=<hostname>. I ran this in the runner sandbox with three manager replicas: each of the six idle VMs on GCE carried the pod name of the replica that created it, and the config on disk still had the literal ${HOSTNAME}.

What are the relevant issue numbers?

• gitlab-com/gl-infra/production-engineering#29652