': URL rejected: Port number was not a decimal number between 0 and 65535

Summary

I use git-submodules in a few projects (self-hosted Gitlab EE). The project URLs are relative to the current project. This used to work very well in the past. Today I tried to run a CI job, and I got the following errors:

fatal: unable to access 'http://gitlab-ci-token:[MASKED]@gitlab/<group>/<project>.git/': URL rejected: Port number was not a decimal number between 0 and 65535

I compared it with older logs and there it worked. The URL is the same. Please note that my runner is using clone_url = "http://gitlab" due to the docker-based setup. I think that using the hostname might be the issue here.

Steps to reproduce

Set up two Git repositories. One should contain the other as a submodule (with a relative path). Add a CI job and try to recursively checkout the Git repository with the submodule.

Example Project

I set up two repositories on gitlab.com, but I cannot reproduce the issue there. I have set up two test repositories at my instance. I#ll try to provide access.

What is the current bug behavior?

Cloning the repository with the submodule fails with

Submodule 'test2' (http://gitlab-ci-token:[MASKED]@gitlab/dleidert/test2.git) registered for path 'test2'
Synchronizing submodule url for 'test2'
Cloning into '/builds/dleidert/test1/test2'...
fatal: unable to access 'http://gitlab-ci-token:[MASKED]@gitlab/dleidert/test2.git/': URL rejected: Port number was not a decimal number between 0 and 65535
fatal: clone of 'http://gitlab-ci-token:[MASKED]@gitlab/dleidert/test2.git' into submodule path '/builds/dleidert/test1/test2' failed
Failed to clone 'test2'. Retry scheduled
Cloning into '/builds/dleidert/test1/test2'...
fatal: unable to access 'http://gitlab-ci-token:[MASKED]@gitlab/dleidert/test2.git/': URL rejected: Port number was not a decimal number between 0 and 65535
fatal: clone of 'http://gitlab-ci-token:[MASKED]@gitlab/dleidert/test2.git' into submodule path '/builds/dleidert/test1/test2' failed
Failed to clone 'test2' a second time, aborting

What is the expected correct behavior?

Cloning should work. Here an older build ~10 months ago, which worked flawlessly:

Submodule '<path>' (http://gitlab-ci-token:[MASKED]@gitlab/<group>/<project>.git) registered for path '<path>'
Synchronizing submodule url for '<path>'
Cloning into '/builds/<group>/<project>/<path>'...

Relevant logs and/or screenshots

Nothing relevant.

Output of checks

n/a

Results of GitLab environment info

GitLab EE 16.3.1 and Gitlab-Runner 16.3.0 (Dockerized setup)

gitlab-rake gitlab:env:info

System information
System:		
Proxy:		no
Current User:	git
Using RVM:	no
Ruby Version:	3.0.6p216
Gem Version:	3.4.18
Bundler Version:2.4.18
Rake Version:	13.0.6
Redis Version:	7.0.12
Sidekiq Version:6.5.7
Go Version:	unknown

GitLab information
Version:	16.3.1-ee
Revision:	67bb799df14
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	13.11
URL:		https://gitlab.wgdd.org
HTTP Clone URL:	https://gitlab.wgdd.org/some-group/some-project.git
SSH Clone URL:	ssh://git@gitlab.wgdd.org:8122/some-group/some-project.git
Elasticsearch:	no
Geo:		no
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: 

GitLab Shell
Version:	14.26.0
Repository storages:
- default: 	unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell

Gitaly
- default Address: 	unix:/var/opt/gitlab/gitaly/gitaly.socket
- default Version: 	16.3.1
- default Git Version: 	2.41.0.gl1

Results of GitLab application Check

All checks are fine.

Possible fixes

I think that the fact that clone_url contains just a hostname and not a FQDN causes the issue. However, I tried quite many solutions: e.g. adding a FQDN-like hostname to the gitlab container and using that in clone_url, or adding the port to clone_url. None of that works, and it used to work out-of-the-box.