Cannot override auto-deploy Variables in some stages

We're using the auto-deploy-image v0.1.0 with a modified deploy script in order to work with a Helm chart monorepo.

Our CI/CD flow is:

• build
• test
• deploy-dev # deploys to the Dev K8s cluster
• deploy-prod # deploys to the Prod K8s cluster

We've overridden the script to allow deploying a specified chart w/in the monorepo w/ custom values (set up as a File variable in the CI settings), and also to allow pre/post scripts as some charts need a bit of configuring w/ Helm initialized.

This flow has worked well so far, but we're now seeing strange inconsistencies in the POST_SCRIPTS between the deploy-dev and deploy-prod stages, where deploy-dev has access to all the overridden variables but in the deploy-prod the variables seem to be set back to theauto-deploy-image defaults.

Here is an example log from the deploy-prod stage, where you can see the POST_SCRIPT doesn't have access to the correct variables:

70 $ if [[ -e "$PRE_SCRIPT" ]]; then # collapsed multi-line command
71 $ VALUES_FILE_VAR_NAME=$(printf "%s" "$RELEASE_NAME"_"$CI_ENVIRONMENT_NAME"_VALUES | tr '-' '_' | awk '{ print toupper($0) }')
72 $ if [[ -z ${!VALUES_FILE_VAR_NAME+x} ]]; then # collapsed multi-line command
73 $ [[ ! -e "$VALUES_FILE" ]] && echo "values file "$VALUES_FILE" does not exist, creating empty." && touch "$VALUES_FILE"
74 $ SUBSTITUTED_VALUES_FILE="$VALUES_FILE"_substituted
75 $ envsubst <"$VALUES_FILE" > "$SUBSTITUTED_VALUES_FILE"
76 $ export CHART="$CHART_DIR"/"$CHART_NAME"
77 $ helm dependency update "$CHART"
78 No requirements found in /builds/fintechstudios/.../charts/stable/document-resolver/charts.
79 $ echo "Releasing $CHART as $RELEASE_NAME with values from $SUBSTITUTED_VALUES_FILE into the namespace $KUBE_NAMESPACE"
80 Releasing charts/stable/document-resolver as doc-ingestion-document-resolver with values from /builds/fintechstudios/.tmp/DOC_INGESTION_DOCUMENT_RESOLVER_PROD_VALUES_substituted into the namespace flink
81 $ helm upgrade --install \ # collapsed multi-line command
82 Release "doc-ingestion-document-resolver" does not exist. Installing it now.
83 NAME:   doc-ingestion-document-resolver
84 LAST DEPLOYED: Mon Jan 13 18:18:33 2020
85 NAMESPACE: flink
86 STATUS: DEPLOYED

...

102 $ if [[ -e "$POST_SCRIPT" ]]; then # collapsed multi-line command
103 Error from server (NotFound): namespaces "document-ingestion-13925771-prod" not found

Here's the custom setup:

.chart-deploy:
  image: registry.gitlab.com/gitlab-org/cluster-integration/auto-deploy-image:v0.1.0
  variables:
    TILLER_NAMESPACE: "tiller"
    HELM_HOST: "localhost:44134"
    KUBE_NAMESPACE: "flink" # override
    CHART_DIR: "charts/stable"
    # PRE_SCRIPT: "" # override to add a script to execute right before the helm install
    # POST_SCRIPT: "" # override to add a script to execute right after the helm install
    # CHART_NAME: "" # override with specific chart name
    # RELEASE_NAME: "" # override with name for release
  only:
    refs:
      - master
#  can use before_script to set up {PRE,POST}_SCRIPT
#  before_script:
  script:
    - apk add --no-cache gettext # for envsubst
    # see: https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image/blob/393bc3a5774ea46285104f1221135a1ecc0c679d/src/bin/auto-deploy
    - auto-deploy ensure_namespace
    # configure helm and local tiller
    - ./scripts/tiller-init.sh # creates the tiller namespace, if not already present
    - nohup ./scripts/tiller-start.sh & # starts tiller locally
    - helm version --debug # check to make sure it can connect
    - source ./scripts/helm-init.sh
    - |
      if [[ -e "$PRE_SCRIPT" ]]; then
        bash "$PRE_SCRIPT"
      fi
    # Values files can be set for each environment in the GitLab CI/CD settings as "$RELEASE_NAME"_"$CI_ENVIRONMENT_NAME"_VALUES in SCREAMING_SNAKE_CASE
    - VALUES_FILE_VAR_NAME=$(printf "%s" "$RELEASE_NAME"_"$CI_ENVIRONMENT_NAME"_VALUES | tr '-' '_' | awk '{ print toupper($0) }')
    # expand the generated values file name, and use a temp values file if one doesn't exist
    - |
      if [[ -z ${!VALUES_FILE_VAR_NAME+x} ]]; then
        echo "values file not provided, using empty."
        VALUES_FILE="values.yaml"
      else
        VALUES_FILE="${!VALUES_FILE_VAR_NAME}"
      fi
    - '[[ ! -e "$VALUES_FILE" ]] && echo "values file "$VALUES_FILE" does not exist, creating empty." && touch "$VALUES_FILE"'
    - SUBSTITUTED_VALUES_FILE="$VALUES_FILE"_substituted
    - envsubst <"$VALUES_FILE" > "$SUBSTITUTED_VALUES_FILE"
    - export CHART="$CHART_DIR"/"$CHART_NAME"
    - helm dependency update "$CHART"
    - echo "Releasing $CHART as $RELEASE_NAME with values from $SUBSTITUTED_VALUES_FILE into the namespace $KUBE_NAMESPACE"
    - |
      helm upgrade --install \
                   --wait \
                   --namespace="$KUBE_NAMESPACE" \
                   --values="$SUBSTITUTED_VALUES_FILE" \
                   $HELM_UPGRADE_EXTRA_ARGS \
                   "$RELEASE_NAME" \
                   "$CHART"
    - |
      if [[ -e "$POST_SCRIPT" ]]; then
        bash "$POST_SCRIPT"
      fi

.chart-deploy-dev: &chart-deploy-dev-template
  extends: .chart-deploy
  stage: deploy-dev
  allow_failure: true
  environment:
    name: dev

.chart-deploy-prod: &chart-deploy-prod-template
  extends: .chart-deploy
  stage: deploy-prod
  allow_failure: false
  when: manual
  environment:
    name: prod


## Then to setting up a chart deployment is as easy as:
.chart-deploy-k8s-dashboard: &chart-deploy-k8s-dashboard-template
  variables:
    KUBE_NAMESPACE: "kube-system"
    CHART_NAME: "k8s-dashboard"
    RELEASE_NAME: "k8s-dashboard"
  only:
    changes:
      - charts/stable/k8s-dashboard/**/*
      - .gitlab-ci.yml

Any idea what could be causing this weird behavior? Thanks!