gitlab-runner with the shell executor sometimes fails to fetch submodules
Summary
gitlab-runner with the shell executor sometimes fails to fetch submodules, with an authentication error, like this:
Synchronizing submodule url for 'subrepo'
remote: HTTP Basic: Access denied
fatal: Authentication failed for 'https://gitlab.aldebaran.lan/agility/subrepo/'
Unable to fetch in submodule path 'subrepo'; trying to directly fetch 8f814d7239d52e9799a5dd45e13803254770a6dc:
remote: HTTP Basic: Access denied
fatal: Authentication failed for 'https://gitlab.aldebaran.lan/agility/subrepo/'
Fetched in submodule path 'subrepo', but it did not contain 8f814d7239d52e9799a5dd45e13803254770a6dc. Direct fetching of that commit failed.
It appears that under some circumstances, when reusing an existing Git repository, gitlab-runner fails to update the submodule url, and thus reuses the url from a previous job, with an outdated CI_JOB_TOKEN, leading to the authentication error.
Steps to reproduce
-
Create two repos (eg. "superrepo" and "subrepo"), each containing an initial commit.
-
setup a runner with shell executor. Tag it (eg with "test_submodule") to ensure it handles all the pipelines we'll trigger.
-
Run the following script, which will
- create three branches:
- ${PREFIX}rel1: with no submodule
- ${PREFIX}rel2: with the "subrepo" submodule, on master
- ${PREFIX}rel3: with the "subrepo" submodule, on ${PREFIX}master
- run six successive pipelines, building those branches:
- ${PREFIX}rel1
- ${PREFIX}rel2
- ${PREFIX}rel1
- ${PREFIX}rel3
- ${PREFIX}rel2
- ${PREFIX}rel3
- create three branches:
set -x
set -e
PREFIX=test3_
git clone git@gitlab.aldebaran.lan:agility/superrepo.git
cd superrepo
git checkout -b ${PREFIX}rel1
cat > .gitlab-ci.yml <<EOF
variables:
GIT_SUBMODULE_STRATEGY: normal
build:
tags:
# use a tag to ensure the same single runner builds all the successive jobs
- test_submodule
script:
- git --version
- test -f .git/config && cat .git/config
- test -f subrepo/.git && cat subrepo/.git
- test -f .gitmodules && cat .gitmodules
- test -f .git/modules/subrepo/config && cat .git/modules/subrepo/config
- test -f .git/modules/subrepo/HEAD && cat .git/modules/subrepo/HEAD
- git config --list --show-origin
EOF
git add .gitlab-ci.yml
git commit -m "setup ci"
# push and trigger a build of rel1
git push
git checkout -b ${PREFIX}rel2
git submodule add ../subrepo
git commit -m "add subrepo"
# push and trigger a build of rel2
git push
# trigger a build of rel1
curl --request POST --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "https://gitlab.aldebaran.lan/api/v4/projects/agility%2Fsuperrepo/pipeline?ref=${PREFIX}rel1"
git checkout -b ${PREFIX}rel3
cd subrepo
git checkout -b ${PREFIX}master
echo "Not much." > README.md
git add README.md
git commit -m "update README"
git push
cd ..
git add subrepo
git commit -m "update subrepo"
# push and trigger a build of rel3 (will fail)
git push
sleep 30
# trigger a build of rel2 (will succeed)
curl --request POST --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "https://gitlab.aldebaran.lan/api/v4/projects/agility%2Fsuperrepo/pipeline?ref=${PREFIX}rel2"
# trigger a build of rel3 (will succeed)
curl --request POST --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "https://gitlab.aldebaran.lan/api/v4/projects/agility%2Fsuperrepo/pipeline?ref=${PREFIX}rel3"
.gitlab-ci.yml
variables:
GIT_SUBMODULE_STRATEGY: normal
build:
tags:
# use a tag to ensure the same single runner builds all the successive jobs
- test_submodule
script:
- git --version
- test -f .git/config && cat .git/config
- test -f subrepo/.git && cat subrepo/.git
- test -f .gitmodules && cat .gitmodules
- test -f .git/modules/subrepo/config && cat .git/modules/subrepo/config
- test -f .git/modules/subrepo/HEAD && cat .git/modules/subrepo/HEAD
- git config --list --show-origin
Actual behavior
Pipeline 4 fails with an authentication error. The 5 others succeed:
- ${PREFIX}rel1 -> success
- ${PREFIX}rel2 -> success
- ${PREFIX}rel1 -> success
- ${PREFIX}rel3 -> failure
- ${PREFIX}rel2 -> success
- ${PREFIX}rel3 -> success
Note that pipeline 4 and 6 (try to) build the exact same commit. The difference in behavior must come from some internal state in the runner local git repo, which is entirely managed by gitlab-runner.
Expected behavior
All pipelines succeed.
Relevant logs and/or screenshots
job log of pipeline 1 (which succeeds)
[0KRunning with gitlab-runner 13.4.0 (4e1f20da)
[0;m[0K on sbarthelemy@ald-1401-de uxEZtquU
[0;msection_start:1600777804:prepare_executor
[0K[0K[36;1mPreparing the "shell" executor[0;m
[0;m[0KUsing Shell executor...
[0;msection_end:1600777804:prepare_executor
[0Ksection_start:1600777804:prepare_script
[0K[0K[36;1mPreparing environment[0;m
[0;mRunning on ald-1401-de...
section_end:1600777804:prepare_script
[0Ksection_start:1600777804:get_sources
[0K[0K[36;1mGetting source from Git repository[0;m
[0;m[32;1mFetching changes with git depth set to 50...[0;m
Reinitialized existing Git repository in /var/cache/gitlab-runner/builds/uxEZtquU/0/agility/superrepo/.git/
[32;1mChecking out 55d52082 as test3_rel1...[0;m
warning: unable to rmdir 'subrepo': Directory not empty
Removing subrepo/
[32;1mUpdating/initializing submodules...[0;m
section_end:1600777805:get_sources
[0Ksection_start:1600777805:step_script
[0K[0K[36;1mExecuting "step_script" stage of the job script[0;m
[0;m[32;1m$ git --version[0;m
git version 2.28.0
[32;1m$ test -f .git/config && cat .git/config[0;m
[fetch]
recurseSubmodules = false
[http "https://gitlab.aldebaran.lan"]
sslCAInfo = /var/cache/gitlab-runner/builds/uxEZtquU/0/agility/superrepo.tmp/CI_SERVER_TLS_CA_FILE
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
[remote "origin"]
url = https://gitlab-ci-token:[MASKED]@gitlab.aldebaran.lan/agility/superrepo.git
fetch = +refs/heads/*:refs/remotes/origin/*
[submodule "subrepo"]
active = true
url = https://gitlab-ci-token:XXXXXX@gitlab.aldebaran.lan/agility/subrepo
[32;1m$ test -f subrepo/.git && cat subrepo/.git[0;m
[32;1m$ test -f .gitmodules && cat .gitmodules[0;m
[32;1m$ test -f .git/modules/subrepo/config && cat .git/modules/subrepo/config[0;m
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
worktree = ../../../subrepo
[remote "origin"]
url = https://gitlab-ci-token:XXXXXX@gitlab.aldebaran.lan/agility/subrepo
fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
remote = origin
merge = refs/heads/master
[32;1m$ test -f .git/modules/subrepo/HEAD && cat .git/modules/subrepo/HEAD[0;m
768b641ed386e1e83866f719b46d83148c9db360
[32;1m$ git config --list --show-origin[0;m
file:/etc/gitconfig filter.lfs.clean=git-lfs clean -- %f
file:/etc/gitconfig filter.lfs.smudge=git-lfs smudge -- %f
file:/etc/gitconfig filter.lfs.process=git-lfs filter-process
file:/etc/gitconfig filter.lfs.required=true
file:.git/config fetch.recursesubmodules=false
file:.git/config http.https://gitlab.aldebaran.lan.sslcainfo=/var/cache/gitlab-runner/builds/uxEZtquU/0/agility/superrepo.tmp/CI_SERVER_TLS_CA_FILE
file:.git/config core.repositoryformatversion=0
file:.git/config core.filemode=true
file:.git/config core.bare=false
file:.git/config core.logallrefupdates=true
file:.git/config remote.origin.url=https://gitlab-ci-token:[MASKED]@gitlab.aldebaran.lan/agility/superrepo.git
file:.git/config remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
file:.git/config submodule.subrepo.active=true
file:.git/config submodule.subrepo.url=https://gitlab-ci-token:XXXXXX@gitlab.aldebaran.lan/agility/subrepo
section_end:1600777805:step_script
[0K[32;1mJob succeeded
[0;m
job log of pipeline 2 (which succeeds)
[0KRunning with gitlab-runner 13.4.0 (4e1f20da)
[0;m[0K on sbarthelemy@ald-1401-de uxEZtquU
[0;msection_start:1600777807:prepare_executor
[0K[0K[36;1mPreparing the "shell" executor[0;m
[0;m[0KUsing Shell executor...
[0;msection_end:1600777807:prepare_executor
[0Ksection_start:1600777807:prepare_script
[0K[0K[36;1mPreparing environment[0;m
[0;mRunning on ald-1401-de...
section_end:1600777807:prepare_script
[0Ksection_start:1600777807:get_sources
[0K[0K[36;1mGetting source from Git repository[0;m
[0;m[32;1mFetching changes with git depth set to 50...[0;m
Reinitialized existing Git repository in /var/cache/gitlab-runner/builds/uxEZtquU/0/agility/superrepo/.git/
[32;1mChecking out 382295f9 as test3_rel2...[0;m
[32;1mUpdating/initializing submodules...[0;m
Synchronizing submodule url for 'subrepo'
Submodule path 'subrepo': checked out '7d4529f6c059029fa172cf54bcb7f77f8cbe1b76'
Entering 'subrepo'
section_end:1600777808:get_sources
[0Ksection_start:1600777808:step_script
[0K[0K[36;1mExecuting "step_script" stage of the job script[0;m
[0;m[32;1m$ git --version[0;m
git version 2.28.0
[32;1m$ test -f .git/config && cat .git/config[0;m
[fetch]
recurseSubmodules = false
[http "https://gitlab.aldebaran.lan"]
sslCAInfo = /var/cache/gitlab-runner/builds/uxEZtquU/0/agility/superrepo.tmp/CI_SERVER_TLS_CA_FILE
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
[remote "origin"]
url = https://gitlab-ci-token:[MASKED]@gitlab.aldebaran.lan/agility/superrepo.git
fetch = +refs/heads/*:refs/remotes/origin/*
[submodule "subrepo"]
active = true
url = https://gitlab-ci-token:[MASKED]@gitlab.aldebaran.lan/agility/subrepo
[32;1m$ test -f subrepo/.git && cat subrepo/.git[0;m
gitdir: ../.git/modules/subrepo
[32;1m$ test -f .gitmodules && cat .gitmodules[0;m
[submodule "subrepo"]
path = subrepo
url = ../subrepo
[32;1m$ test -f .git/modules/subrepo/config && cat .git/modules/subrepo/config[0;m
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
worktree = ../../../subrepo
[remote "origin"]
url = https://gitlab-ci-token:XXXXXX@gitlab.aldebaran.lan/agility/subrepo
fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
remote = origin
merge = refs/heads/master
[32;1m$ test -f .git/modules/subrepo/HEAD && cat .git/modules/subrepo/HEAD[0;m
7d4529f6c059029fa172cf54bcb7f77f8cbe1b76
[32;1m$ git config --list --show-origin[0;m
file:/etc/gitconfig filter.lfs.clean=git-lfs clean -- %f
file:/etc/gitconfig filter.lfs.smudge=git-lfs smudge -- %f
file:/etc/gitconfig filter.lfs.process=git-lfs filter-process
file:/etc/gitconfig filter.lfs.required=true
file:.git/config fetch.recursesubmodules=false
file:.git/config http.https://gitlab.aldebaran.lan.sslcainfo=/var/cache/gitlab-runner/builds/uxEZtquU/0/agility/superrepo.tmp/CI_SERVER_TLS_CA_FILE
file:.git/config core.repositoryformatversion=0
file:.git/config core.filemode=true
file:.git/config core.bare=false
file:.git/config core.logallrefupdates=true
file:.git/config remote.origin.url=https://gitlab-ci-token:[MASKED]@gitlab.aldebaran.lan/agility/superrepo.git
file:.git/config remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
file:.git/config submodule.subrepo.active=true
file:.git/config submodule.subrepo.url=https://gitlab-ci-token:[MASKED]@gitlab.aldebaran.lan/agility/subrepo
section_end:1600777808:step_script
[0K[32;1mJob succeeded
[0;m
job log of pipeline 3 (which succeeds)
[0KRunning with gitlab-runner 13.4.0 (4e1f20da)
[0;m[0K on sbarthelemy@ald-1401-de uxEZtquU
[0;msection_start:1600777810:prepare_executor
[0K[0K[36;1mPreparing the "shell" executor[0;m
[0;m[0KUsing Shell executor...
[0;msection_end:1600777810:prepare_executor
[0Ksection_start:1600777810:prepare_script
[0K[0K[36;1mPreparing environment[0;m
[0;mRunning on ald-1401-de...
section_end:1600777810:prepare_script
[0Ksection_start:1600777810:get_sources
[0K[0K[36;1mGetting source from Git repository[0;m
[0;m[32;1mFetching changes with git depth set to 50...[0;m
Reinitialized existing Git repository in /var/cache/gitlab-runner/builds/uxEZtquU/0/agility/superrepo/.git/
[32;1mChecking out 55d52082 as test3_rel1...[0;m
warning: unable to rmdir 'subrepo': Directory not empty
Removing subrepo/
[32;1mUpdating/initializing submodules...[0;m
section_end:1600777811:get_sources
[0Ksection_start:1600777811:step_script
[0K[0K[36;1mExecuting "step_script" stage of the job script[0;m
[0;m[32;1m$ git --version[0;m
git version 2.28.0
[32;1m$ test -f .git/config && cat .git/config[0;m
[fetch]
recurseSubmodules = false
[http "https://gitlab.aldebaran.lan"]
sslCAInfo = /var/cache/gitlab-runner/builds/uxEZtquU/0/agility/superrepo.tmp/CI_SERVER_TLS_CA_FILE
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
[remote "origin"]
url = https://gitlab-ci-token:[MASKED]@gitlab.aldebaran.lan/agility/superrepo.git
fetch = +refs/heads/*:refs/remotes/origin/*
[submodule "subrepo"]
active = true
url = https://gitlab-ci-token:XXXXXX@gitlab.aldebaran.lan/agility/subrepo
[32;1m$ test -f subrepo/.git && cat subrepo/.git[0;m
[32;1m$ test -f .gitmodules && cat .gitmodules[0;m
[32;1m$ test -f .git/modules/subrepo/config && cat .git/modules/subrepo/config[0;m
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
worktree = ../../../subrepo
[remote "origin"]
url = https://gitlab-ci-token:XXXXXX@gitlab.aldebaran.lan/agility/subrepo
fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
remote = origin
merge = refs/heads/master
[32;1m$ test -f .git/modules/subrepo/HEAD && cat .git/modules/subrepo/HEAD[0;m
7d4529f6c059029fa172cf54bcb7f77f8cbe1b76
[32;1m$ git config --list --show-origin[0;m
file:/etc/gitconfig filter.lfs.clean=git-lfs clean -- %f
file:/etc/gitconfig filter.lfs.smudge=git-lfs smudge -- %f
file:/etc/gitconfig filter.lfs.process=git-lfs filter-process
file:/etc/gitconfig filter.lfs.required=true
file:.git/config fetch.recursesubmodules=false
file:.git/config http.https://gitlab.aldebaran.lan.sslcainfo=/var/cache/gitlab-runner/builds/uxEZtquU/0/agility/superrepo.tmp/CI_SERVER_TLS_CA_FILE
file:.git/config core.repositoryformatversion=0
file:.git/config core.filemode=true
file:.git/config core.bare=false
file:.git/config core.logallrefupdates=true
file:.git/config remote.origin.url=https://gitlab-ci-token:[MASKED]@gitlab.aldebaran.lan/agility/superrepo.git
file:.git/config remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
file:.git/config submodule.subrepo.active=true
file:.git/config submodule.subrepo.url=https://gitlab-ci-token:XXXXXX@gitlab.aldebaran.lan/agility/subrepo
section_end:1600777811:step_script
[0K[32;1mJob succeeded
[0;m
job log of pipeline 4 (which fails)
[0KRunning with gitlab-runner 13.4.0 (4e1f20da)
[0;m[0K on sbarthelemy@ald-1401-de uxEZtquU
[0;msection_start:1600777813:prepare_executor
[0K[0K[36;1mPreparing the "shell" executor[0;m
[0;m[0KUsing Shell executor...
[0;msection_end:1600777813:prepare_executor
[0Ksection_start:1600777813:prepare_script
[0K[0K[36;1mPreparing environment[0;m
[0;mRunning on ald-1401-de...
section_end:1600777813:prepare_script
[0Ksection_start:1600777813:get_sources
[0K[0K[36;1mGetting source from Git repository[0;m
[0;m[32;1mFetching changes with git depth set to 50...[0;m
Reinitialized existing Git repository in /var/cache/gitlab-runner/builds/uxEZtquU/0/agility/superrepo/.git/
[32;1mChecking out 57293d4f as test3_rel3...[0;m
[32;1mUpdating/initializing submodules...[0;m
Synchronizing submodule url for 'subrepo'
remote: HTTP Basic: Access denied
fatal: Authentication failed for 'https://gitlab.aldebaran.lan/agility/subrepo/'
Unable to fetch in submodule path 'subrepo'; trying to directly fetch 8f814d7239d52e9799a5dd45e13803254770a6dc:
remote: HTTP Basic: Access denied
fatal: Authentication failed for 'https://gitlab.aldebaran.lan/agility/subrepo/'
Fetched in submodule path 'subrepo', but it did not contain 8f814d7239d52e9799a5dd45e13803254770a6dc. Direct fetching of that commit failed.
section_end:1600777814:get_sources
[0K[31;1mERROR: Job failed: exit status 1
[0;m
job log of pipeline 5 (which succeeds)
[0KRunning with gitlab-runner 13.4.0 (4e1f20da)
[0;m[0K on sbarthelemy@ald-1401-de uxEZtquU
[0;msection_start:1600777843:prepare_executor
[0K[0K[36;1mPreparing the "shell" executor[0;m
[0;m[0KUsing Shell executor...
[0;msection_end:1600777843:prepare_executor
[0Ksection_start:1600777843:prepare_script
[0K[0K[36;1mPreparing environment[0;m
[0;mRunning on ald-1401-de...
section_end:1600777843:prepare_script
[0Ksection_start:1600777843:get_sources
[0K[0K[36;1mGetting source from Git repository[0;m
[0;m[32;1mFetching changes with git depth set to 50...[0;m
Reinitialized existing Git repository in /var/cache/gitlab-runner/builds/uxEZtquU/0/agility/superrepo/.git/
[32;1mChecking out 382295f9 as test3_rel2...[0;m
[32;1mUpdating/initializing submodules...[0;m
Synchronizing submodule url for 'subrepo'
Entering 'subrepo'
Entering 'subrepo'
HEAD is now at 7d4529f Initial commit
Entering 'subrepo'
section_end:1600777844:get_sources
[0Ksection_start:1600777844:step_script
[0K[0K[36;1mExecuting "step_script" stage of the job script[0;m
[0;m[32;1m$ git --version[0;m
git version 2.28.0
[32;1m$ test -f .git/config && cat .git/config[0;m
[fetch]
recurseSubmodules = false
[http "https://gitlab.aldebaran.lan"]
sslCAInfo = /var/cache/gitlab-runner/builds/uxEZtquU/0/agility/superrepo.tmp/CI_SERVER_TLS_CA_FILE
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
[remote "origin"]
url = https://gitlab-ci-token:[MASKED]@gitlab.aldebaran.lan/agility/superrepo.git
fetch = +refs/heads/*:refs/remotes/origin/*
[submodule "subrepo"]
active = true
url = https://gitlab-ci-token:[MASKED]@gitlab.aldebaran.lan/agility/subrepo
[32;1m$ test -f subrepo/.git && cat subrepo/.git[0;m
gitdir: ../.git/modules/subrepo
[32;1m$ test -f .gitmodules && cat .gitmodules[0;m
[submodule "subrepo"]
path = subrepo
url = ../subrepo
[32;1m$ test -f .git/modules/subrepo/config && cat .git/modules/subrepo/config[0;m
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
worktree = ../../../subrepo
[remote "origin"]
url = https://gitlab-ci-token:[MASKED]@gitlab.aldebaran.lan/agility/subrepo
fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
remote = origin
merge = refs/heads/master
[32;1m$ test -f .git/modules/subrepo/HEAD && cat .git/modules/subrepo/HEAD[0;m
7d4529f6c059029fa172cf54bcb7f77f8cbe1b76
[32;1m$ git config --list --show-origin[0;m
file:/etc/gitconfig filter.lfs.clean=git-lfs clean -- %f
file:/etc/gitconfig filter.lfs.smudge=git-lfs smudge -- %f
file:/etc/gitconfig filter.lfs.process=git-lfs filter-process
file:/etc/gitconfig filter.lfs.required=true
file:.git/config fetch.recursesubmodules=false
file:.git/config http.https://gitlab.aldebaran.lan.sslcainfo=/var/cache/gitlab-runner/builds/uxEZtquU/0/agility/superrepo.tmp/CI_SERVER_TLS_CA_FILE
file:.git/config core.repositoryformatversion=0
file:.git/config core.filemode=true
file:.git/config core.bare=false
file:.git/config core.logallrefupdates=true
file:.git/config remote.origin.url=https://gitlab-ci-token:[MASKED]@gitlab.aldebaran.lan/agility/superrepo.git
file:.git/config remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
file:.git/config submodule.subrepo.active=true
file:.git/config submodule.subrepo.url=https://gitlab-ci-token:[MASKED]@gitlab.aldebaran.lan/agility/subrepo
section_end:1600777844:step_script
[0K[32;1mJob succeeded
[0;m
job log of pipeline 6 (which succeeds)
[0KRunning with gitlab-runner 13.4.0 (4e1f20da)
[0;m[0K on sbarthelemy@ald-1401-de uxEZtquU
[0;msection_start:1600777846:prepare_executor
[0K[0K[36;1mPreparing the "shell" executor[0;m
[0;m[0KUsing Shell executor...
[0;msection_end:1600777846:prepare_executor
[0Ksection_start:1600777846:prepare_script
[0K[0K[36;1mPreparing environment[0;m
[0;mRunning on ald-1401-de...
section_end:1600777846:prepare_script
[0Ksection_start:1600777846:get_sources
[0K[0K[36;1mGetting source from Git repository[0;m
[0;m[32;1mFetching changes with git depth set to 50...[0;m
Reinitialized existing Git repository in /var/cache/gitlab-runner/builds/uxEZtquU/0/agility/superrepo/.git/
[32;1mChecking out 57293d4f as test3_rel3...[0;m
[32;1mUpdating/initializing submodules...[0;m
Synchronizing submodule url for 'subrepo'
Entering 'subrepo'
Entering 'subrepo'
HEAD is now at 7d4529f Initial commit
warning: redirecting to https://gitlab.aldebaran.lan/agility/subrepo.git/
From https://gitlab.aldebaran.lan/agility/subrepo
* [new branch] test3_master -> origin/test3_master
Submodule path 'subrepo': checked out '8f814d7239d52e9799a5dd45e13803254770a6dc'
Entering 'subrepo'
section_end:1600777848:get_sources
[0Ksection_start:1600777848:step_script
[0K[0K[36;1mExecuting "step_script" stage of the job script[0;m
[0;m[32;1m$ git --version[0;m
git version 2.28.0
[32;1m$ test -f .git/config && cat .git/config[0;m
[fetch]
recurseSubmodules = false
[http "https://gitlab.aldebaran.lan"]
sslCAInfo = /var/cache/gitlab-runner/builds/uxEZtquU/0/agility/superrepo.tmp/CI_SERVER_TLS_CA_FILE
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
[remote "origin"]
url = https://gitlab-ci-token:[MASKED]@gitlab.aldebaran.lan/agility/superrepo.git
fetch = +refs/heads/*:refs/remotes/origin/*
[submodule "subrepo"]
active = true
url = https://gitlab-ci-token:[MASKED]@gitlab.aldebaran.lan/agility/subrepo
[32;1m$ test -f subrepo/.git && cat subrepo/.git[0;m
gitdir: ../.git/modules/subrepo
[32;1m$ test -f .gitmodules && cat .gitmodules[0;m
[submodule "subrepo"]
path = subrepo
url = ../subrepo
[32;1m$ test -f .git/modules/subrepo/config && cat .git/modules/subrepo/config[0;m
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
worktree = ../../../subrepo
[remote "origin"]
url = https://gitlab-ci-token:[MASKED]@gitlab.aldebaran.lan/agility/subrepo
fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
remote = origin
merge = refs/heads/master
[32;1m$ test -f .git/modules/subrepo/HEAD && cat .git/modules/subrepo/HEAD[0;m
8f814d7239d52e9799a5dd45e13803254770a6dc
[32;1m$ git config --list --show-origin[0;m
file:/etc/gitconfig filter.lfs.clean=git-lfs clean -- %f
file:/etc/gitconfig filter.lfs.smudge=git-lfs smudge -- %f
file:/etc/gitconfig filter.lfs.process=git-lfs filter-process
file:/etc/gitconfig filter.lfs.required=true
file:.git/config fetch.recursesubmodules=false
file:.git/config http.https://gitlab.aldebaran.lan.sslcainfo=/var/cache/gitlab-runner/builds/uxEZtquU/0/agility/superrepo.tmp/CI_SERVER_TLS_CA_FILE
file:.git/config core.repositoryformatversion=0
file:.git/config core.filemode=true
file:.git/config core.bare=false
file:.git/config core.logallrefupdates=true
file:.git/config remote.origin.url=https://gitlab-ci-token:[MASKED]@gitlab.aldebaran.lan/agility/superrepo.git
file:.git/config remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
file:.git/config submodule.subrepo.active=true
file:.git/config submodule.subrepo.url=https://gitlab-ci-token:[MASKED]@gitlab.aldebaran.lan/agility/subrepo
section_end:1600777848:step_script
[0K[32;1mJob succeeded
[0;m
Environment description
This is a custom installation of GitLab Enterprise Edition 13.3.1-ee.
config.toml contents
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "sbarthelemy@ald-1401-de"
output_limit = 524288
url = "https://gitlab.aldebaran.lan/"
token = "XXXXXXXXXXXXXXXXXXXX"
executor = "shell"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[[runners]]
name = "sbarthelemy@ald-1401-de"
output_limit = 1048576
url = "https://gitlab.aldebaran.lan/"
token = "XXXXXXXXXXXXXXXXXXXX"
executor = "shell"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
Used GitLab Runner version
# gitlab-runner --version
Version: 13.4.0
Git revision: 4e1f20da
Git branch: 13-4-stable
GO version: go1.13.8
Built: 2020-09-18T11:16:05+0000
OS/Arch: linux/amd64
I had the same issue with gitlab-runner version 13.2.2.
Analysis
Some more details.
-
at the end of pipeline 2, the repo is on ${PREFIX}rel2, with
- the submodule mentioned .gitmodules
- the submodule checked out in subrepo/
- the submodule url in .git/config, with the current token
- the submodule gitdir in .git/modules/subrepo/
- the submodule url in .git/modules/subrepo/config, with the current token
-
at the end of pipeline 3, the repo is (back) on ${PREFIX}rel1, with
- the submodule NOT mentioned in .gitmodules
- the submodule NOT checked out
- the submodule url in .git/config, with an outdated token
- the submodule gitdir in .git/modules/subrepo
- the submodule url in .git/modules/subrepo/config, with an outdated token
-
after the pipeline 4 failure, if I log into the runner, I can see
- the submodule mentioned .gitmodules
- the submodule NOT checked out in subrepo/
- the submodule url in .git/config, with the current token
- the submodule gitdir in .git/modules/subrepo/
- the submodule url in .git/modules/subrepo/config, with an outdated token
I don't understand why there is a discrepancy between the submodule url in
.git/config
and in.git/modules/subrepo/config
, but it is probably the crux of this bug. -
for pipeline 5, the local submodule already has the need ref (no need to fetch anything), it may help
-
for pipeline 6, the local repo was on a branch where the submodule was used, it may help.
Investigation log (revised 2023-03-19)
-
Tested the scenarios on GitLab Runner 15.9.1 with two runner workers configured. The first runner worker is configured to use the
shell
executor. The second runner is configured to the thedocker
executor. -
The CI jobs fails consistently when a submodule is configured.
-
To date, I did not identify any viable configuration workarounds to resolve the error.
Submodule 'sub-module-cert-test-subrepo' (https://gitlab-ci-token:[MASKED]@gitlab.com/DarrenEastman/sub-module-cert-test-subrepo) registered for path 'sub-module-cert-test-subrepo'
Synchronizing submodule url for 'sub-module-cert-test-subrepo'
Cloning into '/builds/DarrenEastman/sub-module-cert-test/sub-module-cert-test-subrepo'...
remote: The project you were looking for could not be found or you don't have permission to view it.
fatal: repository 'https://gitlab.com/DarrenEastman/sub-module-cert-test-subrepo/' not found
fatal: clone of 'https://gitlab-ci-token:[MASKED]@gitlab.com/DarrenEastman/sub-module-cert-test-subrepo' into submodule path '/builds/DarrenEastman/sub-module-cert-test/sub-module-cert-test-subrepo' failed
Failed to clone 'sub-module-cert-test-subrepo'. Retry scheduled
Cloning into '/builds/DarrenEastman/sub-module-cert-test/sub-module-cert-test-subrepo'...
remote: The project you were looking for could not be found or you don't have permission to view it.
fatal: repository 'https://gitlab.com/DarrenEastman/sub-module-cert-test-subrepo/' not found
fatal: clone of 'https://gitlab-ci-token:[MASKED]@gitlab.com/DarrenEastman/sub-module-cert-test-subrepo' into submodule path '/builds/DarrenEastman/sub-module-cert-test/sub-module-cert-test-subrepo' failed
Failed to clone 'sub-module-cert-test-subrepo' a second time, aborting
section_end:1679271296:get_sources
[0Ksection_start:1679271296:cleanup_file_variables
[0K[0K[36;1mCleaning up project directory and file based variables[0;m[0;m
section_end:1679271297:cleanup_file_variables
[0K[31;1mERROR: Job failed: exit code 1
[0;m