kubernetes executor caches overwritten variables from gitlab-ci.yml file

Summary

Overwritten variables seems to be cached from one job to another.

Steps to reproduce

Deploy a kubernetes gitlab-runner with the bellow configuration:

metrics_server = "0.0.0.0:9252"
concurrent = 10
check_interval = 30
[[runners]]
  name = "RUNNER_NAME"
  output_limit = 40960
  url = "https://GITURL/ci"
  token = "TOKEN"
  executor = "kubernetes"
  [runners.cache]
    Type = "s3"
    ServerAddress = "s3.amazonaws.com"
    AccessKey = "ACCESS_KEY"
    SecretKey = "SECRET_KEY"
    BucketName = "BUCKET"
    BucketLocation = "AWS_REGION"
    Shared = true
  [runners.kubernetes]
    host = ""
    image = "alpine:latest"
    namespace = "namespace-a"
    namespace_overwrite_allowed = ".*"
    privileged = false
    cpu_limit = "500m"
    memory_limit = "4Gi"
    service_cpu_limit = "250m"
    service_memory_limit = "500Mi"
    helper_cpu_limit = "250m"
    helper_memory_limit = "1Gi"
    cpu_request = "250m"
    memory_request = "1Gi"
    service_cpu_request = "100m"
    service_memory_request = "250Mi"
    helper_cpu_request = "100m"
    helper_memory_request = "500Mi"
    terminationGracePeriodSeconds = 90
    service_account = "tiller"
    service_account_overwrite_allowed = ".*"

Create a gitlab-ci.yml file as follows:

stages:
  - deploy_sandbox_retrofit
  - deploy_sandbox_helm

retrofit:
  stage: deploy_sandbox_retrofit
  image: alpine:latest
  tags:
    - test.ash1
    - kubernetes
  script:
    - echo Hello_World_from_stage_deploy_sandbox_retrofit
    - echo 'This job should use namespace-a namespace and tiller SA'

gitlab_runner:
  stage: deploy_sandbox_helm
  image: alpine:latest
  variables:
    CLUSTER: test.ash1
    KUBERNETES_NAMESPACE_OVERWRITE: kube-system
    KUBERNETES_SERVICE_ACCOUNT_OVERWRITE: default
  tags:
    - test.ash1
    - kubernetes
  script:
    - echo "Hello world from deploy_sandbox_helm"
    - echo "This job should run from kube-system and overwrite ns from namespace-a to kube-system"
    - echo "and overwrite SA from tiller to default"

K8S deployment:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "2"
  creationTimestamp: 2017-07-07T18:43:06Z
  generation: 2
  labels:
    app: wondering-mongoose-gitlab-runner-rbac
    chart: gitlab-runner-rbac-0.3.13
    heritage: Tiller
    release: wondering-mongoose
  name: wondering-mongoose-gitlab-runner-rbac
  namespace: prd357
  resourceVersion: "3305976"
  selfLink: /apis/extensions/v1beta1/namespaces/prd357/deployments/wondering-mongoose-gitlab-runner-rbac
  uid: 1dfa0f9f-6344-11e7-bc41-180373f69c63
spec:
  replicas: 1
  selector:
    matchLabels:
      app: wondering-mongoose-gitlab-runner-rbac
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: wondering-mongoose-gitlab-runner-rbac
    spec:
      containers:
      - command:
        - /usr/bin/dumb-init
        - /entrypoint
        env:
        - name: CI_SERVER_URL
          value: https://URL-XXXX/ci
        - name: REGISTRATION_TOKEN
          valueFrom:
            secretKeyRef:
              key: runner-registration-token
              name: wondering-mongoose-gitlab-runner-rbac
        - name: KUBERNETES_IMAGE
          value: alpine:latest
        - name: KUBERNETES_NAMESPACE
          value: prd357
        - name: KUBERNETES_CPU_LIMIT
          value: 500m
        - name: KUBERNETES_MEMORY_LIMIT
          value: 4Gi
        - name: KUBERNETES_CPU_REQUEST
          value: 250m
        - name: KUBERNETES_MEMORY_REQUEST
          value: 1Gi
        - name: KUBERNETES_SERVICE_CPU_LIMIT
          value: 250m
        - name: KUBERNETES_SERVICE_MEMORY_LIMIT
          value: 500Mi
        - name: KUBERNETES_SERVICE_CPU_REQUEST
          value: 100m
        - name: KUBERNETES_SERVICE_MEMORY_REQUEST
          value: 250Mi
        - name: KUBERNETES_HELPER_CPU_LIMIT
          value: 250m
        - name: KUBERNETES_HELPER_MEMORY_LIMIT
          value: 1Gi
        - name: KUBERNETES_HELPER_CPU_REQUEST
          value: 100m
        - name: KUBERNETES_HELPER_MEMORY_REQUEST
          value: 500Mi
        - name: RUNNER_TAG_LIST
          value: k8s-rbac-test.ash1, kubernetes, beta, test.ash1
        - name: REGISTER_RUN_UNTAGGED
          value: "false"
        - name: RUNNER_NAME
          value: prd357
        - name: KUBERNETES_SERVICE_ACCOUNT
          value: tiller
        - name: CONFIG_FILE_RO
          value: /tmp/gitlab-runner/config.toml
        - name: KUBERNETES_SERVICE_ACCOUNT_OVERWRITE_ALLOWED
          value: .*
        - name: RUNNER_EXECUTOR
          value: kubernetes
        - name: CACHE_SHARED
          value: "true"
        - name: CACHE_TYPE
          value: s3
        - name: KUBERNETES_TERMINATIONGRACEPERIODSECONDS
          value: "90"
        - name: RUNNER_OUTPUT_LIMIT
          value: "40960"
        - name: S3_ACCESS_KEY
          value: ACCESS_KEY
        - name: S3_BUCKET_LOCATION
          value: us-east-1
        - name: S3_BUCKET_NAME
          value: S3_BUCKET_FOO
        - name: S3_SECRET_KEY
          value: SECRET_KEY_XXXXXX
        - name: S3_SERVER_ADDRESS
          value: s3.amazonaws.com
        - name: KUBERNETES_NAMESPACE_OVERWRITE_ALLOWED
          value: .*
        image:  gitlab/gitlab-runner:alpine-v9.2.2
        imagePullPolicy: IfNotPresent
        livenessProbe:
          exec:
            command:
            - /usr/bin/pgrep
            - gitlab-ci-multi
          failureThreshold: 3
          initialDelaySeconds: 60
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: wondering-mongoose-gitlab-runner-rbac
        ports:
        - containerPort: 9252
          name: http-metrics
          protocol: TCP
        readinessProbe:
          exec:
            command:
            - /usr/bin/pgrep
            - gitlab-ci-multi
          failureThreshold: 3
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          limits:
            cpu: 750m
            memory: 2Gi
          requests:
            cpu: 100m
            memory: 128Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /tmp/gitlab-runner
          name: config
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: gitlab-runner
      serviceAccountName: gitlab-runner
      terminationGracePeriodSeconds: 60
      volumes:
      - configMap:
          defaultMode: 420
          items:
          - key: config.toml
            path: config.toml
          name: wondering-mongoose-gitlab-runner-rbac
        name: config
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: 2017-07-07T18:43:07Z
    lastUpdateTime: 2017-07-07T18:43:07Z
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  observedGeneration: 2
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

Actual behavior

Job: retrofit

Running with gitlab-ci-multi-runner 9.2.2 (893d769)
  on namespace-a (914d7be3)
Using Kubernetes namespace: another-namespace
Using Kubernetes executor with image alpine:latest ...
Waiting for pod another-namespace/runner-914d7be3-project-16460-concurrent-0b115t to be running, status is Pending
Waiting for pod another-namespace/runner-914d7be3-project-16460-concurrent-0b115t to be running, status is Pending
Running on runner-914d7be3-project-16460-concurrent-0b115t via wondering-mongoose-gitlab-runner-rbac-705552118-krcwl...
Cloning repository...
Cloning into '/kubernetes/cluster-config'...
Checking out cfbf27fe as dev-jesus...
Skipping Git submodules setup
$ echo Hello_World_from_stage_deploy_sandbox_retrofit
Hello_World_from_stage_deploy_sandbox_retrofit
$ echo 'This job should use namespace-a namespace and tiller SA'
This job should use namespace-a namespace and tiller SA
Job succeeded

Somehow the executor is ignoring that the Namespace should be the default namespace-a, the SA seems to be the correct one. Its overwriting the namespace from another completely different pipeline run, that was overwriting the namespace to another-namespace

Job: gitlab_runner

Running with gitlab-ci-multi-runner 9.2.2 (893d769)
  on namespace-a (914d7be3)
Overwritting configured namespace, from another-namespace to kube-system
Overwritting configured ServiceAccount, from tiller to default
Using Kubernetes namespace: kube-system
Using Kubernetes executor with image alpine:latest ...
Waiting for pod kube-system/runner-914d7be3-project-16460-concurrent-0mb9lq to be running, status is Pending
Waiting for pod kube-system/runner-914d7be3-project-16460-concurrent-0mb9lq to be running, status is Pending
Waiting for pod kube-system/runner-914d7be3-project-16460-concurrent-0mb9lq to be running, status is Pending
Running on runner-914d7be3-project-16460-concurrent-0mb9lq via wondering-mongoose-gitlab-runner-rbac-705552118-krcwl...
Cloning repository...
Cloning into '/kubernetes/cluster-config'...
Checking out cfbf27fe as dev-jesus...
Skipping Git submodules setup
$ echo "Hello world from deploy_sandbox_helm"
Hello world from deploy_sandbox_helm
$ echo "This job should run from kube-system and overwrite ns from namespace-a to kube-system"
This job should run from kube-system and overwrite ns from namespace-a to kube-system
$ echo "and overwrite SA from tiller to default"
and overwrite SA from tiller to default
Job succeeded

Again it looks that the namespace another-namespace got cached somewhere.

Expected behavior

Executor pods should be scheduled according configuration.

Environment description

Tectonic 1.6 Kubernetes cluster, gitlab runner alpine:v9.2.2

Gitlab CE: 9.3.5