Skip to content

Following docs for *base_dir config options causes K8s to reject job pods

Summary

Currently, setting logs_base_dir and scripts_base_dir both to /tmp for example (as the documentation suggests), results in the runner failing to create job pods.

Steps to reproduce

Deploy GitLab runners using the k8s executor. Set runner config options as outlined here

.gitlab-ci.yml
default:
  image: quay.io/helmpack/chart-testing:latest
  tags:
    - homelab

chart:lint:
  variables:
    ALL_CHARTS: ${ALL_CHARTS:-"false"}
    CLUSTER_NAME: ${CI_PIPELINE_ID}-${K8S_VERSION}
    GIT_DEPTH: 50
    GIT_STRATEGY: clone
  before_script:
    - git remote -v
    - git fetch
  script:
    - |
      CHANGED_CHARTS=$(ct list-changed --config .gitlab/ct.yaml)
      CT_FLAGS=(--config .gitlab/ct.yaml --lint-conf .gitlab/lint.yaml --helm-repo-extra-args "wojoinc-charts=--username gitlab-ci-token --password ${CI_JOB_TOKEN}")

      if [[ $ALL_CHARTS == "true" ]]; then
        CT_FLAGS+=(--all)
      fi

      echo "Detected changes in charts:"
      echo "${CHANGED_CHARTS[*]}"

      ct lint "${CT_FLAGS[@]}"
      echo "CHANGED_CHARTS=$CHANGED_CHARTS" >> variables.env
      echo "Printing env file"
      cat variables.env
  artifacts:
    reports:
      dotenv: variables.env
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      when: never
    - if: $CI_COMMIT_TAG
      when: never
    - if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH
      changes:
        paths:
          - charts/*/Chart.yaml

Actual behavior

Instead of producing paths like /tmp/logs-52717018-7386788331 and /tmp/scripts-52717018-7386788331, both volumes are mounted at /tmp-52717018-7386788331 which causes Kubernetes to reject the pod, as two volumes cannot be mounted to the same path.

Expected behavior

Given logs_base_dir and scripts_base_dir set to /tmp,

Volumes should be mounted at /tmp/logs-{CI_PROJECT_ID}-{CI_JOB_ID}, /tmp/scripts-{CI_PROJECT_ID}-{CI_JOB_ID}

Job pods should not be rejected by Kubernetes.

Relevant logs and/or screenshots

image

job log
[0KRunning with gitlab-runner 17.2.0 (6428c288)[0;m
[0K  on gitlab-runner-7ff77d5df5-grs2t 2sm93yd4w, system ID: r_TSwzr5Hiht4J[0;m
[0K  feature flags: FF_USE_ADVANCED_POD_SPEC_CONFIGURATION:true[0;m
section_start:1721422207:prepare_executor
[0K[0K[36;1mPreparing the "kubernetes" executor[0;m[0;m
[0KUsing Kubernetes namespace: gitlab-runner[0;m
[0KUsing Kubernetes executor with image quay.io/helmpack/chart-testing:latest ...[0;m
[0KUsing attach strategy to execute scripts...[0;m
section_end:1721422207:prepare_executor
[0Ksection_start:1721422207:prepare_script
[0K[0K[36;1mPreparing environment[0;m[0;m
[0KUsing FF_USE_POD_ACTIVE_DEADLINE_SECONDS, the Pod activeDeadlineSeconds will be set to the job timeout: 1h0m0s...[0;m
[0;33mWARNING: Advanced Pod Spec configuration enabled, merging the provided PodSpec to the generated one. This is a beta feature and is subject to change. Feedback is collected in this issue: https://gitlab.com/gitlab-org/gitlab-runner/-/issues/29659 ...[0;m
section_end:1721422207:prepare_script
[0K[31;1mERROR: Error cleaning up pod: resource name may not be empty[0;m
[31;1mERROR: Job failed (system failure): prepare environment: setting up build pod: Pod "runner-2sm93yd4w-project-52717018-concurrent-0-x93l2sxa" is invalid: [spec.containers[0].volumeMounts[1].mountPath: Invalid value: "/tmp-52717018-7386788331": must be unique, spec.containers[1].volumeMounts[1].mountPath: Invalid value: "/tmp-52717018-7386788331": must be unique, spec.initContainers[0].volumeMounts[1].mountPath: Invalid value: "/tmp-52717018-7386788331": must be unique]. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information[0;m

Environment description

Kubernetes executor, running on vanilla Kubernetes 1.30.2.

config.toml contents
        [[runners]]
          environment = ["FF_USE_ADVANCED_POD_SPEC_CONFIGURATION=true"]
          builds_dir = "/tmp/builds"
        [runners.kubernetes]
          namespace = "{{.Release.Namespace}}"
          logs_base_dir = "/tmp"
          scripts_base_dir = "/tmp"
          privileged = false
          allowPrivilegeEscalation = false
        [[runners.kubernetes.pod_spec]]
          name = "hostUsers"
          patch = '''
            [{"op": "add", "path": "/hostUsers", "value": false}]
          '''
          patch_type = "json"

Used GitLab Runner version

Running with gitlab-runner 17.2.0 (6428c288)
  on gitlab-runner-7ff77d5df5-grs2t 2sm93yd4w, system ID: r_TSwzr5Hiht4J
  feature flags: FF_USE_ADVANCED_POD_SPEC_CONFIGURATION:true
Preparing the "kubernetes" executor
Using Kubernetes namespace: gitlab-runner
Using Kubernetes executor with image quay.io/helmpack/chart-testing:latest ...
Using attach strategy to execute scripts...

Possible fixes

As a workaround, users can set logs_base_dir and scripts_base_dir to different paths, but this doesn't address documentation or fix the config options to follow their intended functionality.

Additionally, I've opened !4893 (merged) to fix the config options to function as they were intended by !4805 (merged)