Can't set environment variable key as pod label

Overview

For pod labels, we allow variable expansion, for example, if inside of the config.toml of the Runner we have like below, the variable $CI_JOB_ID will be expanded and replaced with the value of CI_JOB_ID from the varaibles.

    [runners.kubernetes.pod_labels]
      job_id = "$CI_JOB_ID"

When a user sets this inside of the values.yml they get the following:

podLabels:
  job_id: "$CI_JOB_ID"

    [runners.kubernetes.pod_labels]
      job_id = ""

This is unexpected behavior because the user would expect CI_JOB_ID to be present, but we are expanding that environment variable during registration. gitlab-org/charts/gitlab-runner!117 (closed) fixes this, however, it is a breaking change since there can be a reason why a user would want it to work like it is right now (to expand during registration)

Workaround

values.yml

# Set the value of `job_id` to be the value of `CI_JOB_ID`
podLabels:
  job_id: "$CI_JOB_ID"

# Set the value of the `CI_JOB_ID` to be `$CI_JOB_ID` which will end up being saved in the Runner config
envVars:
  - name: CI_JOB_ID
    value: "$CI_JOB_ID"

generated config.toml:

    [runners.kubernetes.pod_labels]
      job_id = "$CI_JOB_ID"

Pod running a job with the correct labels:

Labels:       job_id=2085
              pod=runner-pnbw9pre-project-19-concurrent-0