Skip to content

gitlab-runner with the shell executor sometimes fails to fetch submodules

Summary

gitlab-runner with the shell executor sometimes fails to fetch submodules, with an authentication error, like this:

Synchronizing submodule url for 'subrepo'
remote: HTTP Basic: Access denied
fatal: Authentication failed for 'https://gitlab.aldebaran.lan/agility/subrepo/'
Unable to fetch in submodule path 'subrepo'; trying to directly fetch 8f814d7239d52e9799a5dd45e13803254770a6dc:
remote: HTTP Basic: Access denied
fatal: Authentication failed for 'https://gitlab.aldebaran.lan/agility/subrepo/'
Fetched in submodule path 'subrepo', but it did not contain 8f814d7239d52e9799a5dd45e13803254770a6dc. Direct fetching of that commit failed.

It appears that under some circumstances, when reusing an existing Git repository, gitlab-runner fails to update the submodule url, and thus reuses the url from a previous job, with an outdated CI_JOB_TOKEN, leading to the authentication error.

Steps to reproduce

  • Create two repos (eg. "superrepo" and "subrepo"), each containing an initial commit.

  • setup a runner with shell executor. Tag it (eg with "test_submodule") to ensure it handles all the pipelines we'll trigger.

  • Run the following script, which will

    • create three branches:
      • ${PREFIX}rel1: with no submodule
      • ${PREFIX}rel2: with the "subrepo" submodule, on master
      • ${PREFIX}rel3: with the "subrepo" submodule, on ${PREFIX}master
    • run six successive pipelines, building those branches:
      1. ${PREFIX}rel1
      2. ${PREFIX}rel2
      3. ${PREFIX}rel1
      4. ${PREFIX}rel3
      5. ${PREFIX}rel2
      6. ${PREFIX}rel3
set -x
set -e
PREFIX=test3_

git clone git@gitlab.aldebaran.lan:agility/superrepo.git
cd superrepo
git checkout -b ${PREFIX}rel1
cat > .gitlab-ci.yml <<EOF
variables:
  GIT_SUBMODULE_STRATEGY: normal

build:
  tags:
  # use a tag to ensure the same single runner builds all the successive jobs
  - test_submodule
  script:
  - git --version
  - test -f .git/config && cat .git/config
  - test -f subrepo/.git && cat subrepo/.git
  - test -f .gitmodules && cat .gitmodules
  - test -f .git/modules/subrepo/config && cat .git/modules/subrepo/config
  - test -f .git/modules/subrepo/HEAD && cat .git/modules/subrepo/HEAD
  - git config --list --show-origin
EOF

git add .gitlab-ci.yml
git commit -m "setup ci"
# push and trigger a build of rel1
git push

git checkout -b ${PREFIX}rel2
git submodule add ../subrepo
git commit -m "add subrepo"
# push and trigger a build of rel2
git push
# trigger a build of rel1
curl --request POST --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "https://gitlab.aldebaran.lan/api/v4/projects/agility%2Fsuperrepo/pipeline?ref=${PREFIX}rel1"
git checkout -b ${PREFIX}rel3
cd subrepo
git checkout -b ${PREFIX}master
echo "Not much." > README.md
git add README.md
git commit -m "update README"
git push
cd ..
git add subrepo
git commit -m "update subrepo"
# push and trigger a build of rel3 (will fail)
git push
sleep 30
# trigger a build of rel2 (will succeed)
curl --request POST --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "https://gitlab.aldebaran.lan/api/v4/projects/agility%2Fsuperrepo/pipeline?ref=${PREFIX}rel2"
# trigger a build of rel3 (will succeed)
curl --request POST --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "https://gitlab.aldebaran.lan/api/v4/projects/agility%2Fsuperrepo/pipeline?ref=${PREFIX}rel3"
.gitlab-ci.yml 
variables:
  GIT_SUBMODULE_STRATEGY: normal

build:
  tags:
  # use a tag to ensure the same single runner builds all the successive jobs
  - test_submodule
  script:
  - git --version
  - test -f .git/config && cat .git/config
  - test -f subrepo/.git && cat subrepo/.git
  - test -f .gitmodules && cat .gitmodules
  - test -f .git/modules/subrepo/config && cat .git/modules/subrepo/config
  - test -f .git/modules/subrepo/HEAD && cat .git/modules/subrepo/HEAD
  - git config --list --show-origin

Actual behavior

Pipeline 4 fails with an authentication error. The 5 others succeed:

  1. ${PREFIX}rel1 -> success
  2. ${PREFIX}rel2 -> success
  3. ${PREFIX}rel1 -> success
  4. ${PREFIX}rel3 -> failure
  5. ${PREFIX}rel2 -> success
  6. ${PREFIX}rel3 -> success

Note that pipeline 4 and 6 (try to) build the exact same commit. The difference in behavior must come from some internal state in the runner local git repo, which is entirely managed by gitlab-runner.

Expected behavior

All pipelines succeed.

Relevant logs and/or screenshots

job log of pipeline 1 (which succeeds) 
[0KRunning with gitlab-runner 13.4.0 (4e1f20da)
[0;m[0K  on sbarthelemy@ald-1401-de uxEZtquU
[0;msection_start:1600777804:prepare_executor
[0K[0K[36;1mPreparing the "shell" executor[0;m
[0;m[0KUsing Shell executor...
[0;msection_end:1600777804:prepare_executor
[0Ksection_start:1600777804:prepare_script
[0K[0K[36;1mPreparing environment[0;m
[0;mRunning on ald-1401-de...
section_end:1600777804:prepare_script
[0Ksection_start:1600777804:get_sources
[0K[0K[36;1mGetting source from Git repository[0;m
[0;m[32;1mFetching changes with git depth set to 50...[0;m
Reinitialized existing Git repository in /var/cache/gitlab-runner/builds/uxEZtquU/0/agility/superrepo/.git/
[32;1mChecking out 55d52082 as test3_rel1...[0;m
warning: unable to rmdir 'subrepo': Directory not empty
Removing subrepo/

[32;1mUpdating/initializing submodules...[0;m
section_end:1600777805:get_sources
[0Ksection_start:1600777805:step_script
[0K[0K[36;1mExecuting "step_script" stage of the job script[0;m
[0;m[32;1m$ git --version[0;m
git version 2.28.0
[32;1m$ test -f .git/config && cat .git/config[0;m
[fetch]
	recurseSubmodules = false
[http "https://gitlab.aldebaran.lan"]
	sslCAInfo = /var/cache/gitlab-runner/builds/uxEZtquU/0/agility/superrepo.tmp/CI_SERVER_TLS_CA_FILE
[core]
	repositoryformatversion = 0
	filemode = true
	bare = false
	logallrefupdates = true
[remote "origin"]
	url = https://gitlab-ci-token:[MASKED]@gitlab.aldebaran.lan/agility/superrepo.git
	fetch = +refs/heads/*:refs/remotes/origin/*
[submodule "subrepo"]
	active = true
	url = https://gitlab-ci-token:XXXXXX@gitlab.aldebaran.lan/agility/subrepo
[32;1m$ test -f subrepo/.git && cat subrepo/.git[0;m
[32;1m$ test -f .gitmodules && cat .gitmodules[0;m
[32;1m$ test -f .git/modules/subrepo/config && cat .git/modules/subrepo/config[0;m
[core]
	repositoryformatversion = 0
	filemode = true
	bare = false
	logallrefupdates = true
	worktree = ../../../subrepo
[remote "origin"]
	url = https://gitlab-ci-token:XXXXXX@gitlab.aldebaran.lan/agility/subrepo
	fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
	remote = origin
	merge = refs/heads/master
[32;1m$ test -f .git/modules/subrepo/HEAD && cat .git/modules/subrepo/HEAD[0;m
768b641ed386e1e83866f719b46d83148c9db360
[32;1m$ git config --list --show-origin[0;m
file:/etc/gitconfig	filter.lfs.clean=git-lfs clean -- %f
file:/etc/gitconfig	filter.lfs.smudge=git-lfs smudge -- %f
file:/etc/gitconfig	filter.lfs.process=git-lfs filter-process
file:/etc/gitconfig	filter.lfs.required=true
file:.git/config	fetch.recursesubmodules=false
file:.git/config	http.https://gitlab.aldebaran.lan.sslcainfo=/var/cache/gitlab-runner/builds/uxEZtquU/0/agility/superrepo.tmp/CI_SERVER_TLS_CA_FILE
file:.git/config	core.repositoryformatversion=0
file:.git/config	core.filemode=true
file:.git/config	core.bare=false
file:.git/config	core.logallrefupdates=true
file:.git/config	remote.origin.url=https://gitlab-ci-token:[MASKED]@gitlab.aldebaran.lan/agility/superrepo.git
file:.git/config	remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
file:.git/config	submodule.subrepo.active=true
file:.git/config	submodule.subrepo.url=https://gitlab-ci-token:XXXXXX@gitlab.aldebaran.lan/agility/subrepo
section_end:1600777805:step_script
[0K[32;1mJob succeeded
[0;m
job log of pipeline 2 (which succeeds) 
[0KRunning with gitlab-runner 13.4.0 (4e1f20da)
[0;m[0K  on sbarthelemy@ald-1401-de uxEZtquU
[0;msection_start:1600777807:prepare_executor
[0K[0K[36;1mPreparing the "shell" executor[0;m
[0;m[0KUsing Shell executor...
[0;msection_end:1600777807:prepare_executor
[0Ksection_start:1600777807:prepare_script
[0K[0K[36;1mPreparing environment[0;m
[0;mRunning on ald-1401-de...
section_end:1600777807:prepare_script
[0Ksection_start:1600777807:get_sources
[0K[0K[36;1mGetting source from Git repository[0;m
[0;m[32;1mFetching changes with git depth set to 50...[0;m
Reinitialized existing Git repository in /var/cache/gitlab-runner/builds/uxEZtquU/0/agility/superrepo/.git/
[32;1mChecking out 382295f9 as test3_rel2...[0;m

[32;1mUpdating/initializing submodules...[0;m
Synchronizing submodule url for 'subrepo'
Submodule path 'subrepo': checked out '7d4529f6c059029fa172cf54bcb7f77f8cbe1b76'
Entering 'subrepo'
section_end:1600777808:get_sources
[0Ksection_start:1600777808:step_script
[0K[0K[36;1mExecuting "step_script" stage of the job script[0;m
[0;m[32;1m$ git --version[0;m
git version 2.28.0
[32;1m$ test -f .git/config && cat .git/config[0;m
[fetch]
	recurseSubmodules = false
[http "https://gitlab.aldebaran.lan"]
	sslCAInfo = /var/cache/gitlab-runner/builds/uxEZtquU/0/agility/superrepo.tmp/CI_SERVER_TLS_CA_FILE
[core]
	repositoryformatversion = 0
	filemode = true
	bare = false
	logallrefupdates = true
[remote "origin"]
	url = https://gitlab-ci-token:[MASKED]@gitlab.aldebaran.lan/agility/superrepo.git
	fetch = +refs/heads/*:refs/remotes/origin/*
[submodule "subrepo"]
	active = true
	url = https://gitlab-ci-token:[MASKED]@gitlab.aldebaran.lan/agility/subrepo
[32;1m$ test -f subrepo/.git && cat subrepo/.git[0;m
gitdir: ../.git/modules/subrepo
[32;1m$ test -f .gitmodules && cat .gitmodules[0;m
[submodule "subrepo"]
	path = subrepo
	url = ../subrepo
[32;1m$ test -f .git/modules/subrepo/config && cat .git/modules/subrepo/config[0;m
[core]
	repositoryformatversion = 0
	filemode = true
	bare = false
	logallrefupdates = true
	worktree = ../../../subrepo
[remote "origin"]
	url = https://gitlab-ci-token:XXXXXX@gitlab.aldebaran.lan/agility/subrepo
	fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
	remote = origin
	merge = refs/heads/master
[32;1m$ test -f .git/modules/subrepo/HEAD && cat .git/modules/subrepo/HEAD[0;m
7d4529f6c059029fa172cf54bcb7f77f8cbe1b76
[32;1m$ git config --list --show-origin[0;m
file:/etc/gitconfig	filter.lfs.clean=git-lfs clean -- %f
file:/etc/gitconfig	filter.lfs.smudge=git-lfs smudge -- %f
file:/etc/gitconfig	filter.lfs.process=git-lfs filter-process
file:/etc/gitconfig	filter.lfs.required=true
file:.git/config	fetch.recursesubmodules=false
file:.git/config	http.https://gitlab.aldebaran.lan.sslcainfo=/var/cache/gitlab-runner/builds/uxEZtquU/0/agility/superrepo.tmp/CI_SERVER_TLS_CA_FILE
file:.git/config	core.repositoryformatversion=0
file:.git/config	core.filemode=true
file:.git/config	core.bare=false
file:.git/config	core.logallrefupdates=true
file:.git/config	remote.origin.url=https://gitlab-ci-token:[MASKED]@gitlab.aldebaran.lan/agility/superrepo.git
file:.git/config	remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
file:.git/config	submodule.subrepo.active=true
file:.git/config	submodule.subrepo.url=https://gitlab-ci-token:[MASKED]@gitlab.aldebaran.lan/agility/subrepo
section_end:1600777808:step_script
[0K[32;1mJob succeeded
[0;m
job log of pipeline 3 (which succeeds) 
[0KRunning with gitlab-runner 13.4.0 (4e1f20da)
[0;m[0K  on sbarthelemy@ald-1401-de uxEZtquU
[0;msection_start:1600777810:prepare_executor
[0K[0K[36;1mPreparing the "shell" executor[0;m
[0;m[0KUsing Shell executor...
[0;msection_end:1600777810:prepare_executor
[0Ksection_start:1600777810:prepare_script
[0K[0K[36;1mPreparing environment[0;m
[0;mRunning on ald-1401-de...
section_end:1600777810:prepare_script
[0Ksection_start:1600777810:get_sources
[0K[0K[36;1mGetting source from Git repository[0;m
[0;m[32;1mFetching changes with git depth set to 50...[0;m
Reinitialized existing Git repository in /var/cache/gitlab-runner/builds/uxEZtquU/0/agility/superrepo/.git/
[32;1mChecking out 55d52082 as test3_rel1...[0;m
warning: unable to rmdir 'subrepo': Directory not empty
Removing subrepo/

[32;1mUpdating/initializing submodules...[0;m
section_end:1600777811:get_sources
[0Ksection_start:1600777811:step_script
[0K[0K[36;1mExecuting "step_script" stage of the job script[0;m
[0;m[32;1m$ git --version[0;m
git version 2.28.0
[32;1m$ test -f .git/config && cat .git/config[0;m
[fetch]
	recurseSubmodules = false
[http "https://gitlab.aldebaran.lan"]
	sslCAInfo = /var/cache/gitlab-runner/builds/uxEZtquU/0/agility/superrepo.tmp/CI_SERVER_TLS_CA_FILE
[core]
	repositoryformatversion = 0
	filemode = true
	bare = false
	logallrefupdates = true
[remote "origin"]
	url = https://gitlab-ci-token:[MASKED]@gitlab.aldebaran.lan/agility/superrepo.git
	fetch = +refs/heads/*:refs/remotes/origin/*
[submodule "subrepo"]
	active = true
	url = https://gitlab-ci-token:XXXXXX@gitlab.aldebaran.lan/agility/subrepo
[32;1m$ test -f subrepo/.git && cat subrepo/.git[0;m
[32;1m$ test -f .gitmodules && cat .gitmodules[0;m
[32;1m$ test -f .git/modules/subrepo/config && cat .git/modules/subrepo/config[0;m
[core]
	repositoryformatversion = 0
	filemode = true
	bare = false
	logallrefupdates = true
	worktree = ../../../subrepo
[remote "origin"]
	url = https://gitlab-ci-token:XXXXXX@gitlab.aldebaran.lan/agility/subrepo
	fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
	remote = origin
	merge = refs/heads/master
[32;1m$ test -f .git/modules/subrepo/HEAD && cat .git/modules/subrepo/HEAD[0;m
7d4529f6c059029fa172cf54bcb7f77f8cbe1b76
[32;1m$ git config --list --show-origin[0;m
file:/etc/gitconfig	filter.lfs.clean=git-lfs clean -- %f
file:/etc/gitconfig	filter.lfs.smudge=git-lfs smudge -- %f
file:/etc/gitconfig	filter.lfs.process=git-lfs filter-process
file:/etc/gitconfig	filter.lfs.required=true
file:.git/config	fetch.recursesubmodules=false
file:.git/config	http.https://gitlab.aldebaran.lan.sslcainfo=/var/cache/gitlab-runner/builds/uxEZtquU/0/agility/superrepo.tmp/CI_SERVER_TLS_CA_FILE
file:.git/config	core.repositoryformatversion=0
file:.git/config	core.filemode=true
file:.git/config	core.bare=false
file:.git/config	core.logallrefupdates=true
file:.git/config	remote.origin.url=https://gitlab-ci-token:[MASKED]@gitlab.aldebaran.lan/agility/superrepo.git
file:.git/config	remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
file:.git/config	submodule.subrepo.active=true
file:.git/config	submodule.subrepo.url=https://gitlab-ci-token:XXXXXX@gitlab.aldebaran.lan/agility/subrepo
section_end:1600777811:step_script
[0K[32;1mJob succeeded
[0;m
job log of pipeline 4 (which fails) 
[0KRunning with gitlab-runner 13.4.0 (4e1f20da)
[0;m[0K  on sbarthelemy@ald-1401-de uxEZtquU
[0;msection_start:1600777813:prepare_executor
[0K[0K[36;1mPreparing the "shell" executor[0;m
[0;m[0KUsing Shell executor...
[0;msection_end:1600777813:prepare_executor
[0Ksection_start:1600777813:prepare_script
[0K[0K[36;1mPreparing environment[0;m
[0;mRunning on ald-1401-de...
section_end:1600777813:prepare_script
[0Ksection_start:1600777813:get_sources
[0K[0K[36;1mGetting source from Git repository[0;m
[0;m[32;1mFetching changes with git depth set to 50...[0;m
Reinitialized existing Git repository in /var/cache/gitlab-runner/builds/uxEZtquU/0/agility/superrepo/.git/
[32;1mChecking out 57293d4f as test3_rel3...[0;m

[32;1mUpdating/initializing submodules...[0;m
Synchronizing submodule url for 'subrepo'
remote: HTTP Basic: Access denied
fatal: Authentication failed for 'https://gitlab.aldebaran.lan/agility/subrepo/'
Unable to fetch in submodule path 'subrepo'; trying to directly fetch 8f814d7239d52e9799a5dd45e13803254770a6dc:
remote: HTTP Basic: Access denied
fatal: Authentication failed for 'https://gitlab.aldebaran.lan/agility/subrepo/'
Fetched in submodule path 'subrepo', but it did not contain 8f814d7239d52e9799a5dd45e13803254770a6dc. Direct fetching of that commit failed.
section_end:1600777814:get_sources
[0K[31;1mERROR: Job failed: exit status 1
[0;m
job log of pipeline 5 (which succeeds) 
[0KRunning with gitlab-runner 13.4.0 (4e1f20da)
[0;m[0K  on sbarthelemy@ald-1401-de uxEZtquU
[0;msection_start:1600777843:prepare_executor
[0K[0K[36;1mPreparing the "shell" executor[0;m
[0;m[0KUsing Shell executor...
[0;msection_end:1600777843:prepare_executor
[0Ksection_start:1600777843:prepare_script
[0K[0K[36;1mPreparing environment[0;m
[0;mRunning on ald-1401-de...
section_end:1600777843:prepare_script
[0Ksection_start:1600777843:get_sources
[0K[0K[36;1mGetting source from Git repository[0;m
[0;m[32;1mFetching changes with git depth set to 50...[0;m
Reinitialized existing Git repository in /var/cache/gitlab-runner/builds/uxEZtquU/0/agility/superrepo/.git/
[32;1mChecking out 382295f9 as test3_rel2...[0;m

[32;1mUpdating/initializing submodules...[0;m
Synchronizing submodule url for 'subrepo'
Entering 'subrepo'
Entering 'subrepo'
HEAD is now at 7d4529f Initial commit
Entering 'subrepo'
section_end:1600777844:get_sources
[0Ksection_start:1600777844:step_script
[0K[0K[36;1mExecuting "step_script" stage of the job script[0;m
[0;m[32;1m$ git --version[0;m
git version 2.28.0
[32;1m$ test -f .git/config && cat .git/config[0;m
[fetch]
	recurseSubmodules = false
[http "https://gitlab.aldebaran.lan"]
	sslCAInfo = /var/cache/gitlab-runner/builds/uxEZtquU/0/agility/superrepo.tmp/CI_SERVER_TLS_CA_FILE
[core]
	repositoryformatversion = 0
	filemode = true
	bare = false
	logallrefupdates = true
[remote "origin"]
	url = https://gitlab-ci-token:[MASKED]@gitlab.aldebaran.lan/agility/superrepo.git
	fetch = +refs/heads/*:refs/remotes/origin/*
[submodule "subrepo"]
	active = true
	url = https://gitlab-ci-token:[MASKED]@gitlab.aldebaran.lan/agility/subrepo
[32;1m$ test -f subrepo/.git && cat subrepo/.git[0;m
gitdir: ../.git/modules/subrepo
[32;1m$ test -f .gitmodules && cat .gitmodules[0;m
[submodule "subrepo"]
	path = subrepo
	url = ../subrepo
[32;1m$ test -f .git/modules/subrepo/config && cat .git/modules/subrepo/config[0;m
[core]
	repositoryformatversion = 0
	filemode = true
	bare = false
	logallrefupdates = true
	worktree = ../../../subrepo
[remote "origin"]
	url = https://gitlab-ci-token:[MASKED]@gitlab.aldebaran.lan/agility/subrepo
	fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
	remote = origin
	merge = refs/heads/master
[32;1m$ test -f .git/modules/subrepo/HEAD && cat .git/modules/subrepo/HEAD[0;m
7d4529f6c059029fa172cf54bcb7f77f8cbe1b76
[32;1m$ git config --list --show-origin[0;m
file:/etc/gitconfig	filter.lfs.clean=git-lfs clean -- %f
file:/etc/gitconfig	filter.lfs.smudge=git-lfs smudge -- %f
file:/etc/gitconfig	filter.lfs.process=git-lfs filter-process
file:/etc/gitconfig	filter.lfs.required=true
file:.git/config	fetch.recursesubmodules=false
file:.git/config	http.https://gitlab.aldebaran.lan.sslcainfo=/var/cache/gitlab-runner/builds/uxEZtquU/0/agility/superrepo.tmp/CI_SERVER_TLS_CA_FILE
file:.git/config	core.repositoryformatversion=0
file:.git/config	core.filemode=true
file:.git/config	core.bare=false
file:.git/config	core.logallrefupdates=true
file:.git/config	remote.origin.url=https://gitlab-ci-token:[MASKED]@gitlab.aldebaran.lan/agility/superrepo.git
file:.git/config	remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
file:.git/config	submodule.subrepo.active=true
file:.git/config	submodule.subrepo.url=https://gitlab-ci-token:[MASKED]@gitlab.aldebaran.lan/agility/subrepo
section_end:1600777844:step_script
[0K[32;1mJob succeeded
[0;m
job log of pipeline 6 (which succeeds) 
[0KRunning with gitlab-runner 13.4.0 (4e1f20da)
[0;m[0K  on sbarthelemy@ald-1401-de uxEZtquU
[0;msection_start:1600777846:prepare_executor
[0K[0K[36;1mPreparing the "shell" executor[0;m
[0;m[0KUsing Shell executor...
[0;msection_end:1600777846:prepare_executor
[0Ksection_start:1600777846:prepare_script
[0K[0K[36;1mPreparing environment[0;m
[0;mRunning on ald-1401-de...
section_end:1600777846:prepare_script
[0Ksection_start:1600777846:get_sources
[0K[0K[36;1mGetting source from Git repository[0;m
[0;m[32;1mFetching changes with git depth set to 50...[0;m
Reinitialized existing Git repository in /var/cache/gitlab-runner/builds/uxEZtquU/0/agility/superrepo/.git/
[32;1mChecking out 57293d4f as test3_rel3...[0;m

[32;1mUpdating/initializing submodules...[0;m
Synchronizing submodule url for 'subrepo'
Entering 'subrepo'
Entering 'subrepo'
HEAD is now at 7d4529f Initial commit
warning: redirecting to https://gitlab.aldebaran.lan/agility/subrepo.git/
From https://gitlab.aldebaran.lan/agility/subrepo
 * [new branch]      test3_master -> origin/test3_master
Submodule path 'subrepo': checked out '8f814d7239d52e9799a5dd45e13803254770a6dc'
Entering 'subrepo'
section_end:1600777848:get_sources
[0Ksection_start:1600777848:step_script
[0K[0K[36;1mExecuting "step_script" stage of the job script[0;m
[0;m[32;1m$ git --version[0;m
git version 2.28.0
[32;1m$ test -f .git/config && cat .git/config[0;m
[fetch]
	recurseSubmodules = false
[http "https://gitlab.aldebaran.lan"]
	sslCAInfo = /var/cache/gitlab-runner/builds/uxEZtquU/0/agility/superrepo.tmp/CI_SERVER_TLS_CA_FILE
[core]
	repositoryformatversion = 0
	filemode = true
	bare = false
	logallrefupdates = true
[remote "origin"]
	url = https://gitlab-ci-token:[MASKED]@gitlab.aldebaran.lan/agility/superrepo.git
	fetch = +refs/heads/*:refs/remotes/origin/*
[submodule "subrepo"]
	active = true
	url = https://gitlab-ci-token:[MASKED]@gitlab.aldebaran.lan/agility/subrepo
[32;1m$ test -f subrepo/.git && cat subrepo/.git[0;m
gitdir: ../.git/modules/subrepo
[32;1m$ test -f .gitmodules && cat .gitmodules[0;m
[submodule "subrepo"]
	path = subrepo
	url = ../subrepo
[32;1m$ test -f .git/modules/subrepo/config && cat .git/modules/subrepo/config[0;m
[core]
	repositoryformatversion = 0
	filemode = true
	bare = false
	logallrefupdates = true
	worktree = ../../../subrepo
[remote "origin"]
	url = https://gitlab-ci-token:[MASKED]@gitlab.aldebaran.lan/agility/subrepo
	fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
	remote = origin
	merge = refs/heads/master
[32;1m$ test -f .git/modules/subrepo/HEAD && cat .git/modules/subrepo/HEAD[0;m
8f814d7239d52e9799a5dd45e13803254770a6dc
[32;1m$ git config --list --show-origin[0;m
file:/etc/gitconfig	filter.lfs.clean=git-lfs clean -- %f
file:/etc/gitconfig	filter.lfs.smudge=git-lfs smudge -- %f
file:/etc/gitconfig	filter.lfs.process=git-lfs filter-process
file:/etc/gitconfig	filter.lfs.required=true
file:.git/config	fetch.recursesubmodules=false
file:.git/config	http.https://gitlab.aldebaran.lan.sslcainfo=/var/cache/gitlab-runner/builds/uxEZtquU/0/agility/superrepo.tmp/CI_SERVER_TLS_CA_FILE
file:.git/config	core.repositoryformatversion=0
file:.git/config	core.filemode=true
file:.git/config	core.bare=false
file:.git/config	core.logallrefupdates=true
file:.git/config	remote.origin.url=https://gitlab-ci-token:[MASKED]@gitlab.aldebaran.lan/agility/superrepo.git
file:.git/config	remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
file:.git/config	submodule.subrepo.active=true
file:.git/config	submodule.subrepo.url=https://gitlab-ci-token:[MASKED]@gitlab.aldebaran.lan/agility/subrepo
section_end:1600777848:step_script
[0K[32;1mJob succeeded
[0;m

Environment description

This is a custom installation of GitLab Enterprise Edition 13.3.1-ee.

config.toml contents 
concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "sbarthelemy@ald-1401-de"
  output_limit = 524288
  url = "https://gitlab.aldebaran.lan/"
  token = "XXXXXXXXXXXXXXXXXXXX"
  executor = "shell"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]

[[runners]]
  name = "sbarthelemy@ald-1401-de"
  output_limit = 1048576
  url = "https://gitlab.aldebaran.lan/"
  token = "XXXXXXXXXXXXXXXXXXXX"
  executor = "shell"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]

Used GitLab Runner version

# gitlab-runner --version
Version:      13.4.0
Git revision: 4e1f20da
Git branch:   13-4-stable
GO version:   go1.13.8
Built:        2020-09-18T11:16:05+0000
OS/Arch:      linux/amd64

I had the same issue with gitlab-runner version 13.2.2.

Analysis

Some more details.

  • at the end of pipeline 2, the repo is on ${PREFIX}rel2, with

    • the submodule mentioned .gitmodules
    • the submodule checked out in subrepo/
    • the submodule url in .git/config, with the current token
    • the submodule gitdir in .git/modules/subrepo/
    • the submodule url in .git/modules/subrepo/config, with the current token

  • at the end of pipeline 3, the repo is (back) on ${PREFIX}rel1, with

    • the submodule NOT mentioned in .gitmodules
    • the submodule NOT checked out
    • the submodule url in .git/config, with an outdated token
    • the submodule gitdir in .git/modules/subrepo
    • the submodule url in .git/modules/subrepo/config, with an outdated token

  • after the pipeline 4 failure, if I log into the runner, I can see

    • the submodule mentioned .gitmodules
    • the submodule NOT checked out in subrepo/
    • the submodule url in .git/config, with the current token
    • the submodule gitdir in .git/modules/subrepo/
    • the submodule url in .git/modules/subrepo/config, with an outdated token

    I don't understand why there is a discrepancy between the submodule url in .git/config and in .git/modules/subrepo/config, but it is probably the crux of this bug.

  • for pipeline 5, the local submodule already has the need ref (no need to fetch anything), it may help

  • for pipeline 6, the local repo was on a branch where the submodule was used, it may help.

Investigation log (revised 2023-03-19)

  • Tested the scenarios on GitLab Runner 15.9.1 with two runner workers configured. The first runner worker is configured to use the shell executor. The second runner is configured to the the docker executor.

  • The CI jobs fails consistently when a submodule is configured.

  • To date, I did not identify any viable configuration workarounds to resolve the error.


Submodule 'sub-module-cert-test-subrepo' (https://gitlab-ci-token:[MASKED]@gitlab.com/DarrenEastman/sub-module-cert-test-subrepo) registered for path 'sub-module-cert-test-subrepo'
Synchronizing submodule url for 'sub-module-cert-test-subrepo'
Cloning into '/builds/DarrenEastman/sub-module-cert-test/sub-module-cert-test-subrepo'...
remote: The project you were looking for could not be found or you don't have permission to view it.
fatal: repository 'https://gitlab.com/DarrenEastman/sub-module-cert-test-subrepo/' not found
fatal: clone of 'https://gitlab-ci-token:[MASKED]@gitlab.com/DarrenEastman/sub-module-cert-test-subrepo' into submodule path '/builds/DarrenEastman/sub-module-cert-test/sub-module-cert-test-subrepo' failed
Failed to clone 'sub-module-cert-test-subrepo'. Retry scheduled
Cloning into '/builds/DarrenEastman/sub-module-cert-test/sub-module-cert-test-subrepo'...
remote: The project you were looking for could not be found or you don't have permission to view it.
fatal: repository 'https://gitlab.com/DarrenEastman/sub-module-cert-test-subrepo/' not found
fatal: clone of 'https://gitlab-ci-token:[MASKED]@gitlab.com/DarrenEastman/sub-module-cert-test-subrepo' into submodule path '/builds/DarrenEastman/sub-module-cert-test/sub-module-cert-test-subrepo' failed
Failed to clone 'sub-module-cert-test-subrepo' a second time, aborting
section_end:1679271296:get_sources
[0Ksection_start:1679271296:cleanup_file_variables
[0K[0K[36;1mCleaning up project directory and file based variables[0;m[0;m
section_end:1679271297:cleanup_file_variables
[0K[31;1mERROR: Job failed: exit code 1
[0;m
Edited by Darren Eastman