Random 403 using git submodule on CI/CD

Summary

In a project, we declared a git submodule (https://docs.gitlab.com/ee/ci/git_submodules.html)

When running pipeline, the built-in process to clone the project fails randomly with a HTTP 403.

Steps to reproduce

We are not able to reproduce the bug, this happen randomly and not often. When it happens, we only have to retry the job and then it succeed. Each time it fails, it succeeds the next time.

This can happen with a pipeline triggered by a tag, or a merge in master; there is no particular situation when it fails.

Example Project

This is basic project with a .gitmodules

What is the current bug behavior?

Unable to clone a git submodule, resulting with error : The requested URL returned error: 403

What is the expected correct behavior?

Cloning project is done.

Relevant logs and/or screenshots

Output logs when it fails:

Getting source from Git repository
00:47
Fetching changes...
Initialized empty Git repository in /builds/xxx/services/portal/.git/
Created fresh repository.
Checking out 3894ee44 as 23.5.1...
Updating/initializing submodules recursively...
Submodule 'portal-i18n' (https://gitlab-ci-token:[MASKED]@git.xxx.com/xxx/portal-i18n.git) registered for path 'portal-i18n'
Synchronizing submodule url for 'portal-i18n'
Cloning into '/builds/xxx/services/portal/portal-i18n'...
remote: You are not allowed to download code from this project.
fatal: unable to access 'https://git.xxx.com/xxx/portal-i18n.git/': The requested URL returned error: 403
fatal: clone of 'https://gitlab-ci-token:[MASKED]@git.xxx.com/xxx/portal-i18n.git' into submodule path '/builds/xxx/services/portal/portal-i18n' failed
Failed to clone 'portal-i18n'. Retry scheduled
Cloning into '/builds/xxx/services/portal/portal-i18n'...
remote: You are not allowed to download code from this project.
fatal: unable to access 'https://git.xxx.com/xxx/portal-i18n.git/': The requested URL returned error: 403
fatal: clone of 'https://gitlab-ci-token:[MASKED]@git.xxx.com/xxx/portal-i18n.git' into submodule path '/builds/xxx/services/portal/portal-i18n' failed
Failed to clone 'portal-i18n' a second time, aborting
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: command terminated with exit code 1

Output logs when it succeeds:

Getting source from Git repository
00:52
Fetching changes...
Initialized empty Git repository in /builds/xxx/services/portal/.git/
Created fresh repository.
Checking out 3894ee44 as 23.5.1...
Updating/initializing submodules recursively...
Submodule 'portal-i18n' (https://gitlab-ci-token:[MASKED]@git.xxx.com/xxx/portal-i18n.git) registered for path 'portal-i18n'
Synchronizing submodule url for 'portal-i18n'
Cloning into '/builds/xxx/services/portal/portal-i18n'...
Submodule path 'portal-i18n': checked out 'cd5da9853ffbf3e572dd509f8ca1c3f96da6bc19'
Entering 'portal-i18n'
Entering 'portal-i18n'
Restoring cache
00:17
Checking cache for f8b1304c0272fd17b3bd748925a426ad4b223ec4-58-non_protected...
Downloading cache.zip from https://xxx-gitlab-ci-cache-243ygd9q.s3.dualstack.eu-west-1.amazonaws.com/cache/project/18/f8b1304c0272fd17b3bd748925a426ad4b223ec4-58-non_protected 
Successfully extracted cache
Executing "step_script" stage of the job script
...

There is not relevant logs on runner side, last logs when it fails are :

[32;1mCreated fresh repository.[0;m
Initialized empty Git repository in /builds/xxx/services/portal/.git/

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

System information
System:		Ubuntu 20.04
Proxy:		HTTPS_PROXY: http://proxy.svc.tools.infra.xxx.eu:3128
		no_proxy: 127.0.0.1,localhost,169.254.169.254,elb.amazonaws.com,es.amazonaws.com,git.xxx.com
		https_proxy: http://xxx.com:3128
		NO_PROXY: 127.0.0.1,localhost,169.254.169.254,elb.amazonaws.com,es.amazonaws.com,git.xxx.com
		HTTP_PROXY: http://xxx.com:3128
		http_proxy: http://xxx.com:3128
Current User:	git
Using RVM:	no
Ruby Version:	2.7.7p221
Gem Version:	3.1.6
Bundler Version:2.3.15
Rake Version:	13.0.6
Redis Version:	6.2.8
Sidekiq Version:6.5.7
Go Version:	unknown

GitLab information
Version:	15.9.3-ee
Revision:	d3356768b48
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	12.14
URL:		https://git.xxx.com
HTTP Clone URL:	https://git.xxx.com/some-group/some-project.git
SSH Clone URL:	git@git.xxx.com:some-group/some-project.git
Elasticsearch:	yes
Geo:		no
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: 

GitLab Shell
Version:	14.17.0
Repository storages:
- default: 	tcp://gitaly1.xxx.com:8075
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell

Results of GitLab application Check

Expand for output related to the GitLab application check

Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 14.17.0 ? ... OK (14.17.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/4
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Cable config exists? ... yes
Resque config exists? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ...
15/5 ... yes
16/6 ... yes
51/7 ... yes
16/8 ... yes
7/11 ... yes
7/12 ... yes
30/13 ... yes
30/14 ... yes
38/16 ... yes
7/17 ... yes
51/18 ... yes
51/19 ... yes
51/20 ... yes
9/22 ... yes
30/23 ... yes
30/24 ... yes
16/25 ... yes
7/26 ... yes
30/27 ... yes
51/28 ... yes
7/29 ... yes
7/32 ... yes
7/33 ... yes
7/34 ... yes
2/36 ... yes
15/37 ... yes
13/38 ... yes
7/39 ... yes
7/40 ... yes
7/41 ... yes
21/42 ... yes
7/43 ... yes
16/45 ... yes
7/46 ... yes
7/47 ... yes
7/49 ... yes
16/50 ... yes
9/51 ... yes
15/52 ... yes
9/53 ... yes
20/54 ... yes
51/55 ... yes
20/56 ... yes
20/57 ... yes
20/59 ... yes
20/61 ... yes
16/62 ... yes
9/64 ... yes
21/66 ... yes
20/67 ... yes
21/69 ... yes
20/72 ... yes
20/73 ... yes
51/74 ... yes
51/75 ... yes
20/76 ... yes
9/78 ... yes
7/82 ... yes
8/83 ... yes
7/84 ... yes
9/85 ... yes
51/86 ... yes
51/88 ... yes
51/89 ... yes
51/90 ... yes
7/91 ... yes
13/93 ... yes
20/96 ... yes
20/97 ... yes
16/98 ... yes
51/100 ... yes
21/101 ... yes
16/102 ... yes
11/105 ... yes
20/106 ... yes
20/107 ... yes
37/108 ... yes
16/109 ... yes
7/110 ... yes
19/111 ... yes
16/112 ... yes
51/114 ... yes
20/115 ... yes
16/116 ... yes
51/117 ... yes
7/118 ... yes
9/120 ... yes
51/121 ... yes
16/124 ... yes
16/125 ... yes
15/126 ... yes
35/127 ... yes
11/128 ... yes
20/129 ... yes
20/130 ... yes
38/131 ... yes
37/132 ... yes
21/133 ... yes
8/134 ... yes
7/135 ... yes
37/137 ... yes
38/138 ... yes
38/141 ... yes
7/142 ... yes
51/143 ... yes
2/144 ... yes
51/145 ... yes
35/146 ... yes
30/147 ... yes
39/148 ... yes
38/149 ... yes
7/150 ... yes
7/151 ... yes
7/152 ... yes
20/153 ... yes
7/156 ... yes
11/157 ... yes
16/158 ... yes
9/159 ... yes
38/160 ... yes
16/161 ... yes
16/162 ... yes
7/163 ... yes
38/164 ... yes
2/165 ... yes
51/166 ... yes
51/167 ... yes
20/168 ... yes
9/169 ... yes
7/170 ... yes
20/172 ... yes
20/173 ... yes
13/174 ... yes
37/175 ... yes
7/176 ... yes
20/177 ... yes
7/178 ... yes
35/179 ... yes
35/180 ... yes
7/181 ... yes
45/183 ... yes
20/184 ... yes
11/185 ... yes
40/186 ... yes
37/187 ... yes
11/188 ... yes
47/190 ... yes
47/191 ... yes
47/193 ... yes
8/196 ... yes
47/197 ... yes
16/198 ... yes
7/199 ... yes
47/200 ... yes
20/201 ... yes
47/202 ... yes
20/204 ... yes
37/205 ... yes
38/206 ... yes
20/207 ... yes
37/208 ... yes
20/209 ... yes
48/211 ... yes
51/212 ... yes
51/214 ... yes
32/215 ... yes
37/216 ... yes
37/217 ... yes
47/218 ... yes
16/219 ... yes
37/220 ... yes
54/221 ... yes
53/222 ... yes
16/224 ... yes
41/225 ... yes
55/226 ... yes
16/227 ... yes
47/228 ... yes
56/229 ... yes
56/230 ... yes
56/231 ... yes
56/232 ... yes
40/233 ... yes
8/234 ... yes
8/235 ... yes
11/236 ... yes
8/237 ... yes
8/240 ... yes
37/241 ... yes
11/242 ... yes
56/243 ... yes
8/244 ... yes
54/245 ... yes
36/246 ... yes
8/247 ... yes
47/249 ... yes
20/250 ... yes
7/251 ... yes
8/252 ... yes
36/253 ... yes
36/254 ... yes
5/256 ... yes
37/257 ... yes
36/258 ... yes
21/259 ... yes
16/260 ... yes
5/261 ... yes
9/263 ... yes
20/264 ... yes
39/268 ... yes
11/269 ... yes
35/270 ... yes
20/273 ... yes
21/274 ... yes
11/275 ... yes
73/276 ... yes
7/277 ... yes
11/278 ... yes
13/279 ... yes
38/280 ... yes
21/281 ... yes
23/282 ... yes
81/283 ... yes
7/284 ... yes
487/285 ... yes
7/286 ... yes
20/287 ... yes
38/288 ... yes
107/289 ... yes
88/290 ... yes
64/291 ... yes
7/292 ... yes
64/293 ... yes
23/294 ... yes
20/295 ... yes
11/296 ... yes
7/297 ... yes
71/298 ... yes
100/299 ... yes
86/301 ... yes
86/302 ... yes
20/304 ... yes
89/305 ... yes
107/306 ... yes
11/308 ... yes
11/309 ... yes
38/310 ... yes
108/312 ... yes
107/313 ... yes
80/314 ... yes
39/316 ... yes
110/317 ... yes
40/318 ... yes
21/319 ... yes
64/320 ... yes
11/321 ... yes
35/322 ... yes
35/323 ... yes
80/325 ... yes
9/326 ... yes
38/327 ... yes
11/328 ... yes
11/329 ... yes
11/330 ... yes
128/331 ... yes
38/333 ... yes
48/334 ... yes
128/335 ... yes
128/336 ... yes
80/337 ... yes
55/338 ... yes
40/339 ... yes
7/340 ... yes
155/341 ... yes
108/342 ... yes
478/345 ... yes
128/346 ... yes
16/347 ... yes
164/348 ... yes
487/349 ... yes
108/350 ... yes
20/351 ... yes
94/352 ... yes
20/353 ... yes
20/355 ... yes
478/357 ... yes
20/359 ... yes
478/360 ... yes
20/361 ... yes
38/362 ... yes
492/363 ... yes
107/364 ... yes
14/366 ... yes
108/370 ... yes
108/371 ... yes
14/373 ... yes
148/374 ... yes
492/375 ... yes
164/376 ... yes
21/377 ... yes
108/378 ... yes
521/379 ... yes
37/380 ... yes
92/381 ... yes
40/382 ... yes
36/383 ... yes
32/385 ... yes
14/386 ... yes
478/388 ... yes
547/389 ... yes
71/390 ... yes
155/391 ... yes
155/392 ... yes
547/393 ... yes
562/394 ... yes
547/395 ... yes
38/396 ... yes
547/397 ... yes
40/398 ... yes
11/399 ... yes
20/400 ... yes
35/401 ... yes
38/402 ... yes
35/403 ... yes
594/404 ... yes
547/405 ... yes
612/406 ... yes
612/407 ... yes
128/408 ... yes
128/411 ... yes
38/412 ... yes
478/414 ... yes
478/416 ... yes
84/417 ... yes
492/420 ... yes
20/421 ... yes
576/422 ... yes
2/423 ... yes
2/424 ... yes
20/425 ... yes
113/428 ... yes
80/429 ... yes
35/430 ... yes
108/431 ... yes
108/433 ... yes
516/434 ... yes
487/435 ... yes
487/436 ... yes
487/437 ... yes
175/438 ... yes
492/440 ... yes
35/441 ... yes
679/442 ... yes
108/443 ... yes
478/444 ... yes
51/445 ... yes
80/446 ... yes
487/447 ... yes
487/448 ... yes
Redis version >= 6.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.7)
Git user has default SSH configuration? ... yes
Active users: ... 110
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Elasticsearch version 7.x-8.x or OpenSearch version 1.x ... yes (opensearch 1.3.2)
All migrations must be finished before doing a major upgrade ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished

Possible fixes

None, just retry the job.