Sometimes there is an error with submodule authorisation

Summary

Sometimes when running a pipeline with a submodule and setting the 'GIT_SUBMODULE_STRATEGY' variable to 'recursive', there is a problem accessing the submodule. After restarting the job everything works fine.

Steps to reproduce

1. Create 2 projects
2. In one project configure submodule for second project. In '.gitmodules' file set relative url, example:

[submodule "proj2"]
	path = proj2
	url = ../proj2.git

3. In first project create job with variable GIT_SUBMODULE_STRATEGY set to 'recursive'. Unfortunately, I do not know why there is an error with the authorisation. It happens every X job.

Example Project

I will try to reproduce the error here: group https://gitlab.com/issue-389055

What is the current bug behavior?

When synchronising a submodule, the following message appears:

Synchronizing submodule url for 'proj2'
remote: HTTP Basic: Access denied. The provided password or token is incorrect or your account has 2FA enabled and you must use a personal access token instead of a password. See https://company.com/help/topics/git/troubleshooting_git#error-on-git-fetch-http-basic-access-denied
fatal: Authentication failed for 'https://company.com/devops/szablony/terraspace-m2-dev.git/'
Unable to fetch in submodule path 'proj2'; trying to directly fetch 1d19b9b6fe4c5798cb88c4078d83e91d31eccb15:
remote: HTTP Basic: Access denied. The provided password or token is incorrect or your account has 2FA enabled and you must use a personal access token instead of a password. See https://company.com/help/topics/git/troubleshooting_git#error-on-git-fetch-http-basic-access-denied
fatal: Authentication failed for 'https://company.com/devops/szablony/terraspace-m2-dev.git/'
Fetched in submodule path 'proj2', but it did not contain 1d19b9b6fe4c5798cb88c4078d83e91d31eccb15. Direct fetching of that commit failed.
Cleaning up project directory and file based variables
00:00
ERROR: Job failed: exit code 1

What is the expected correct behavior?

The submodule is successfully cloned

Updating/initializing submodules recursively with git depth set to 20...[0;m
Synchronizing submodule url for 'proj2'
Entering 'proj2'
Entering 'proj2'
HEAD is now at ab44eff fix: startup script and script 052
From https://company.com/proj2
   ab44eff..1d19b9b  main       -> origin/main
Submodule path 'proj2': checked out '1d19b9b6fe4c5798cb88c4078d83e91d31eccb15'
Entering 'proj2'
Entering 'proj2'
section_end:1674554180:get_sources
[0Ksection_start:1674554180:step_script

Relevant logs and/or screenshots

Added earlier

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

root@acc7ae8ad949:/# gitlab-rake gitlab:env:info

System information
System:		
Current User:	git
Using RVM:	no
Ruby Version:	2.7.7p221
Gem Version:	3.1.6
Bundler Version:2.3.15
Rake Version:	13.0.6
Redis Version:	6.2.8
Sidekiq Version:6.5.7
Go Version:	unknown

GitLab information
Version:	15.7.5
Revision:	358d690d91c
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	12.12
URL:		https://company.com
HTTP Clone URL:	https://company.com/some-group/some-project.git
SSH Clone URL:	git@company.com:some-group/some-project.git
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: google_oauth2

GitLab Shell
Version:	14.14.0
Repository storages:
- default: 	unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell

Results of GitLab application Check

Expand for output related to the GitLab application check

root@acc7ae8ad949:/# gitlab-rake gitlab:check SANITIZE=true
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 14.14.0 ? ... OK (14.14.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Cable config exists? ... yes
Resque config exists? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... no
Try fixing it:
sudo chown -R git /var/opt/gitlab/gitlab-rails/uploads
sudo find /var/opt/gitlab/gitlab-rails/uploads -type f -exec chmod 0644 {} ;
sudo find /var/opt/gitlab/gitlab-rails/uploads -type d -not -path /var/opt/gitlab/gitlab-rails/uploads -exec chmod 0700 {} ;
For more information see:
doc/install/installation.md in section "GitLab"
Please fix the error above and rerun the checks.
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ...
5/2 ... yes
11/3 ... yes
11/4 ... yes
5/5 ... yes
11/6 ... yes
15/7 ... yes
17/8 ... yes
12/9 ... yes
12/10 ... yes
21/11 ... yes
22/12 ... yes
12/13 ... yes
12/14 ... yes
11/15 ... yes
11/16 ... yes
27/17 ... yes
11/18 ... yes
29/19 ... yes
26/20 ... yes
26/21 ... yes
26/22 ... yes
26/23 ... yes
26/24 ... yes
26/25 ... yes
17/26 ... yes
11/27 ... yes
32/28 ... yes
33/29 ... yes
33/30 ... yes
33/32 ... yes
11/33 ... yes
33/35 ... yes
2/36 ... yes
11/37 ... yes
41/38 ... yes
33/39 ... yes
27/40 ... yes
11/41 ... yes
50/42 ... yes
52/43 ... yes
11/45 ... yes
17/46 ... yes
5/48 ... yes
5/49 ... yes
11/50 ... yes
27/51 ... yes
11/53 ... yes
11/55 ... yes
5/56 ... yes
67/58 ... yes
27/59 ... yes
68/60 ... yes
69/61 ... yes
11/62 ... yes
2/63 ... yes
80/64 ... yes
11/65 ... yes
11/66 ... yes
50/67 ... yes
17/68 ... yes
87/69 ... yes
17/70 ... yes
17/71 ... yes
5/72 ... yes
11/73 ... yes
11/74 ... yes
87/76 ... yes
11/77 ... yes
89/78 ... yes
89/79 ... yes
89/80 ... yes
11/81 ... yes
89/82 ... yes
11/83 ... yes
89/84 ... yes
89/85 ... yes
89/86 ... yes
11/87 ... yes
100/89 ... yes
33/90 ... yes
103/91 ... yes
103/92 ... yes
105/93 ... yes
89/94 ... yes
89/95 ... yes
89/97 ... yes
112/105 ... yes
2/107 ... yes
112/109 ... yes
112/110 ... yes
112/112 ... yes
105/114 ... yes
105/115 ... yes
115/116 ... yes
115/117 ... yes
112/118 ... yes
115/119 ... yes
105/120 ... yes
119/121 ... yes
112/122 ... yes
11/123 ... yes
112/124 ... yes
105/125 ... yes
27/126 ... yes
27/127 ... yes
2/128 ... yes
132/131 ... yes
130/132 ... yes
130/133 ... yes
130/134 ... yes
130/135 ... yes
27/136 ... yes
27/140 ... yes
27/141 ... yes
132/142 ... yes
15/147 ... yes
136/149 ... yes
136/150 ... yes
130/151 ... yes
105/152 ... yes
106/153 ... yes
130/154 ... yes
139/155 ... yes
130/156 ... yes
130/157 ... yes
139/158 ... yes
130/159 ... yes
140/161 ... yes
144/162 ... yes
151/163 ... yes
144/164 ... yes
144/165 ... yes
146/166 ... yes
146/167 ... yes
150/169 ... yes
150/170 ... yes
140/171 ... yes
151/172 ... yes
158/173 ... yes
158/174 ... yes
158/175 ... yes
158/176 ... yes
158/177 ... yes
158/178 ... yes
154/179 ... yes
154/180 ... yes
156/183 ... yes
156/184 ... yes
158/185 ... yes
158/186 ... yes
154/187 ... yes
154/188 ... yes
154/189 ... yes
156/190 ... yes
146/191 ... yes
11/192 ... yes
11/193 ... yes
154/194 ... yes
154/195 ... yes
154/196 ... yes
158/197 ... yes
22/198 ... yes
22/199 ... yes
151/200 ... yes
158/201 ... yes
154/202 ... yes
11/203 ... yes
178/204 ... yes
174/205 ... yes
178/206 ... yes
174/207 ... yes
174/208 ... yes
178/209 ... yes
178/210 ... yes
174/211 ... yes
174/212 ... yes
154/213 ... yes
158/214 ... yes
154/215 ... yes
144/216 ... yes
161/217 ... yes
178/221 ... yes
181/222 ... yes
181/223 ... yes
151/224 ... yes
22/225 ... yes
150/226 ... yes
158/227 ... yes
150/228 ... yes
11/229 ... yes
158/230 ... yes
158/231 ... yes
17/232 ... yes
17/233 ... yes
105/234 ... yes
105/235 ... yes
158/236 ... yes
188/237 ... yes
188/238 ... yes
158/239 ... yes
22/240 ... yes
190/241 ... yes
190/242 ... yes
105/243 ... yes
11/244 ... yes
154/245 ... yes
154/246 ... yes
11/247 ... yes
154/248 ... yes
158/249 ... yes
196/250 ... yes
196/251 ... yes
198/252 ... yes
198/253 ... yes
174/254 ... yes
174/255 ... yes
190/256 ... yes
190/257 ... yes
151/258 ... yes
154/259 ... yes
174/260 ... yes
5/261 ... yes
154/262 ... yes
202/263 ... yes
202/264 ... yes
5/265 ... yes
158/266 ... yes
158/267 ... yes
115/268 ... yes
203/269 ... yes
158/270 ... yes
178/271 ... yes
158/272 ... yes
27/273 ... yes
158/274 ... yes
215/275 ... yes
215/276 ... yes
181/277 ... yes
158/278 ... yes
154/279 ... yes
11/280 ... yes
154/281 ... yes
228/282 ... yes
228/283 ... yes
154/285 ... yes
27/286 ... yes
228/287 ... yes
158/288 ... yes
158/290 ... yes
198/291 ... yes
198/292 ... yes
228/293 ... yes
228/294 ... yes
228/295 ... yes
228/296 ... yes
228/299 ... yes
181/300 ... yes
236/301 ... yes
236/302 ... yes
240/303 ... yes
240/304 ... yes
156/305 ... yes
154/306 ... yes
154/307 ... yes
181/309 ... yes
158/310 ... yes
158/311 ... yes
240/313 ... yes
240/314 ... yes
240/315 ... yes
5/316 ... yes
5/317 ... yes
5/318 ... yes
541/320 ... yes
541/321 ... yes
158/322 ... yes
547/323 ... yes
547/324 ... yes
158/325 ... yes
547/326 ... yes
547/327 ... yes
228/329 ... yes
158/330 ... yes
11/331 ... yes
27/332 ... yes
158/333 ... yes
588/334 ... yes
226/335 ... yes
593/336 ... yes
593/337 ... yes
154/338 ... yes
547/340 ... yes
547/341 ... yes
207/342 ... yes
605/343 ... yes
158/344 ... yes
593/345 ... yes
617/347 ... yes
617/348 ... yes
240/349 ... yes
240/350 ... yes
612/383 ... yes
612/384 ... yes
158/385 ... yes
158/386 ... yes
593/387 ... yes
593/388 ... yes
593/389 ... yes
677/390 ... yes
677/391 ... yes
240/392 ... yes
240/393 ... yes
158/394 ... yes
154/395 ... yes
154/396 ... yes
158/397 ... yes
178/398 ... yes
Redis version >= 6.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.7)
Git user has default SSH configuration? ... yes
Active users: ... 102
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished

Possible fixes

I tried to debug the problem, but I noticed that on Gitlab.com the project is always initialised from scratch. Locally, I use runners in docker. When I switched GIT_STRATEGY to clone the problem stopped. Presumably the authentication problem occurs when the runner tries to update an existing repository . By default GIT_STRATEGY is set to fetch.