Kubernetes executor cannot clone git submodule

Summary

I have the following configuration:

A gitlab runner running inside an EC2 instance configured to use the docker executor.
A kubernetes cluster with the gitlab-runner helm package instaled

In a job that has to download a git submodule, the kubernetes executor fails saying:

Cloning into '/builds/dataops/Rewards-system/terraform'...
remote: The project you were looking for could not be found or you don't have permission to view it.
fatal: repository 'https://gitlab.onpremise.com/devops/terraform.git/' not found
fatal: clone of 'https://gitlab-ci-token:[MASKED]@gitlab.onpremise.com/devops/terraform.git' into submodule path '/builds/dataops/Rewards-system/terraform' failed
Failed to clone 'terraform' a second time, aborting

We have no special token configuration, I think I might try to configure a token and change the project permissions but I think if this is permission related then both runners should fail

Steps to reproduce

Allow the job to run in either the EC2 gitlab runner with docker executor and in the kubernetes executor. The one in the EC2 with docker executor will pass The kubernetes executor will throw an error during the submodule fetch.

.gitlab-ci.yml

terraform:plan:
  stage: build
  image:
    name: hashicorp/terraform:light
    entrypoint: [""]
  tags:
    - tf
  variables:
    GIT_SUBMODULE_STRATEGY: recursive
    GIT_DEPTH: "0"
  before_script:
      - terraform --version
      - apk --no-cache add git
      - git submodule update --init --recursive
  script:
    - cd terraform/main/apps/rewards/$RELEASE_ENV
    - terraform init -input=false
    - terraform plan -input=false
  rules:
    - if: '(
      $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "dev" ||
      $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "test" ||
      $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "main"
      )
      && $CI_PIPELINE_SOURCE == "merge_request_event"
      '
      variables:
        RELEASE_ENV: "$CI_MERGE_REQUEST_TARGET_BRANCH_NAME"
    - if: '$CI_COMMIT_BRANCH == "dev" || $CI_COMMIT_BRANCH == "test" || $CI_COMMIT_BRANCH == "main"'
      variables:
          RELEASE_ENV: "$CI_COMMIT_BRANCH"

Actual behavior

Kubernetes executor fails. Docker executor passes

Expected behavior

Either both should fail if this is related to permissions or both should pass.

Relevant logs and/or screenshots

job log

Running with gitlab-runner 16.1.0 (b72e108d)
  on gitlab-ci-runner-gitlab-runner-78f5cf6c64-rcbcg EWj75SVmX, system ID: r_NSJZsiqiEnJu
Resolving secrets
00:00
Preparing the "kubernetes" executor
00:00
Using Kubernetes namespace: gitlab-ci
Using Kubernetes executor with image hashicorp/terraform:light ...
Using attach strategy to execute scripts...
Preparing environment
00:04
Waiting for pod gitlab-ci/runner-ewj75svmx-project-80-concurrent-2m4gv2 to be running, status is Pending
Running on runner-ewj75svmx-project-80-concurrent-2m4gv2 via gitlab-ci-runner-gitlab-runner-78f5cf6c64-rcbcg...
Getting source from Git repository
00:02
Fetching changes...
Initialized empty Git repository in /builds/dataops/Rewards-system/.git/
Created fresh repository.
Checking out 0191a806 as detached HEAD (ref is refs/merge-requests/87/head)...
Updating/initializing submodules recursively...
Submodule 'terraform' (https://gitlab-ci-token:[MASKED]@gitlab.onpremise.com/devops/terraform.git) registered for path 'terraform'
Synchronizing submodule url for 'terraform'
Cloning into '/builds/dataops/Rewards-system/terraform'...
remote: The project you were looking for could not be found or you don't have permission to view it.
fatal: repository 'https://gitlab.onpremise.com/devops/terraform.git/' not found
fatal: clone of 'https://gitlab-ci-token:[MASKED]@gitlab.onpremise.com/devops/terraform.git' into submodule path '/builds/dataops/Rewards-system/terraform' failed
Failed to clone 'terraform'. Retry scheduled
Cloning into '/builds/dataops/Rewards-system/terraform'...
remote: The project you were looking for could not be found or you don't have permission to view it.
fatal: repository 'https://gitlab.onpremise.com/devops/terraform.git/' not found
fatal: clone of 'https://gitlab-ci-token:[MASKED]@gitlab.onpremise.com/devops/terraform.git' into submodule path '/builds/dataops/Rewards-system/terraform' failed
Failed to clone 'terraform' a second time, aborting
Cleaning up project directory and file based variables
00:00
ERROR: Job failed: command terminated with exit code 1

Running with gitlab-runner 16.1.0 (b72e108d)
  on APP-GITLAB-RUNNER-01.onpremise.com Ps73VxKx4, system ID: s_7c5da24fdb7d
Resolving secrets
00:00
Preparing the "docker" executor
00:02
Using Docker executor with image hashicorp/terraform:light ...
Pulling docker image hashicorp/terraform:light ...
Using docker image sha256:22ae929e925c6bdb60846c05fbee4f15ef15bd02d101e23f34dd6be69cace0f3 for hashicorp/terraform:light with digest hashicorp/terraform@sha256:2734886211a482abaff7efbbfd8bdb32295c9c35dda124092d46f39b69f47c42 ...
Preparing environment
00:01
Running on runner-ps73vxkx4-project-80-concurrent-0 via APP-GITLAB-RUNNER-01.onpremise.com...
Getting source from Git repository
00:06
Fetching changes...
Reinitialized existing Git repository in /builds/dataops/Rewards-system/.git/
Checking out 0191a806 as detached HEAD (ref is refs/merge-requests/87/head)...
Removing front/build/
Removing front/node_modules/
Updating/initializing submodules recursively...
Synchronizing submodule url for 'terraform'
Entering 'terraform'
Entering 'terraform'
HEAD is now at 881b613 Merge branch 'change-rewards-url' into 'master'
Entering 'terraform'
Entering 'terraform'
Executing "step_script" stage of the job script
... proceeds to continue with the job

Environment description

Kubernetes executor: Running with gitlab-runner 16.1.0 ( b72e108d)

EC2 with docker executor :Running with gitlab-runner 16.1.0 ( b72e108d)
 Client: Docker Engine - Community
 Version:    24.0.4
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.19.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 7
 Server Version: 24.0.4
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
 runc version: v1.1.7-0-g860f061
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
 Kernel Version: 5.15.0-1039-aws
 Operating System: Ubuntu 20.04.6 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 7.749GiB
 Name: APP-GITLAB-RUNNER-01.onpremise.com
 ID: 360db892-214b-4f1a-af78-0ddc6703468c
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

config.toml contents

concurrent = 1
check_interval = 0
shutdown_timeout = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "APP-GITLAB-RUNNER-01.onpremise.com"
  url = "https://gitlab.onpremise.com"
  id = 144
  token = "xxxxxxxxxN6ZB"
  token_obtained_at = 2023-07-11T04:59:52Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "python:3.9"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache", "/certs/client"]
    shm_size = 0

  config: |
    [[runners]]
      environment = [
        "DOCKER_HOST=tcp://docker:2376",
        "DOCKER_TLS_CERTDIR=/certs",
        "DOCKER_TLS_VERIFY=1",
        "DOCKER_CERT_PATH=$DOCKER_TLS_CERTDIR/client"
      ]
      [runners.kubernetes]
        namespace = "{{.Release.Namespace}}"
        image = "ubuntu:20.04"
        cpu_request = "0.5"
        cpu_limit = "1"
        helper_cpu_request = "50m"
        helper_cpu_limit = "500m"
        service_cpu_request = "0.5"
        service_cpu_limit = "1"
        memory_request = "2Gi"
        memory_limit = "3Gi"
        helper_memory_request = "128Mi" 
        helper_memory_limit = "2Gi"
        service_memory_request = "1Gi"
        service_memory_limit = "2Gi"
        privileged = true
        service_account = "{{.Release.Name}}-{{.Chart.Name}}"
        [[runners.kubernetes.volumes.empty_dir]]
          name = "docker-certs"
          mount_path = "/certs/client"
          medium = "Memory"

Used GitLab Runner version

Running with gitlab-runner 16.1.0 (b72e108d)
  on APP-GITLAB-RUNNER-01.onpremise.com Ps73VxKx4, system ID: s_7c5da24fdb7d

Running with gitlab-runner 16.1.0 (b72e108d)
  on gitlab-ci-runner-gitlab-runner-78f5cf6c64-rcbcg EWj75SVmX, system ID: r_NSJZsiqiEnJu

Possible fixes

Edited Sep 18, 2023 by Axel von Bertoldi