Custom clone_url breaks submodule authentication with insteadOf rules

Problem

When a custom url or clone_url is configured in the runner, submodule cloning fails with authentication errors even though the main repository clones successfully. The issue occurs because the git insteadOf rules generated for credential injection only apply to the exact main repository URL, not to submodule URLs from the same domain.

Error Example 

fatal: could not read Username for 'https://git.mydomain.com': terminal prompts disabled
fatal: clone of 'https://git.mydomain.com/marcolz/test-module' into submodule path '/builds/marcolz/test-main/submodules/test-module' failed

Root Cause

The setupExternalGitConfig() method in shells/abstract.go creates insteadOf rules that map the full repository URL with credentials to the canonical URL:

url.https://gitlab-ci-token:[MASKED]@git.mydomain.com/marcolz/test-main.git.insteadOf = https://git.mydomain.com/marcolz/test-main.git

However, submodules reference different repository paths (e.g., https://git.mydomain.com/marcolz/test-module), which don't match this rule. Git then attempts to clone without credentials, resulting in authentication failures.

Proposed Solution

Add an additional base domain insteadOf rule that applies to all URLs under the same domain. This catch-all rule would enable credential injection for any submodule URL from the same domain:

url.https://gitlab-ci-token:[MASKED]@git.mydomain.com.insteadOf = https://git.mydomain.com

Implementation Details

The fix mirrors the logic already present in build_url_helper.go's GetInsteadOfs() method (see MR !5912 (merged)), which handles SSH/Git protocol URL rewrites for submodules when GitSubmoduleForceHTTPS is enabled.

In shells/abstract.go, the setupExternalGitConfig() method should be updated to:

  1. Extract the base URL (scheme + host) from the full repository URL
  2. Generate an additional insteadOf rule mapping the base URL with credentials to the base URL without credentials
  3. This ensures both the main repo and all submodules from the same domain can authenticate

Proposed Diff 

// In setupExternalGitConfig(), after processing the full-path insteadOf rules:

// Add base domain insteadOf rule for submodules
baseURLWithCreds, err := url.Parse(withCreds)
if err == nil && baseURLWithCreds.Host != "" {
    // Extract base URL (scheme + host only)
    baseWithCreds := fmt.Sprintf("%s://%s", baseURLWithCreds.Scheme, baseURLWithCreds.Host)
    baseWithoutCreds := fmt.Sprintf("%s://%s", baseURLWithCreds.Scheme, baseURLWithCreds.Host)
    
    // Replace host with auth-injected version
    if baseURLWithCreds.User != nil {
        baseWithCreds = fmt.Sprintf("%s://%s@%s", baseURLWithCreds.Scheme, baseURLWithCreds.User.String(), baseURLWithCreds.Host)
    }
    
    insteadOfs = append(insteadOfs, [2]string{baseWithCreds, baseWithoutCreds})
}

Related Files

  • shells/abstract.go - setupExternalGitConfig() method
  • common/build_url_helper.go - GetInsteadOfs() method (reference implementation from MR !5912 (merged))
  • MR !5912 (merged) - "Externalize git configuration" (merged, contains similar logic)

Testing

Test with:

  • A private repository with custom clone_url configured
  • Submodules from the same domain (both private and public)
  • Both FF_GIT_URLS_WITHOUT_TOKENS disabled and enabled
  • Verify submodule cloning succeeds with proper authentication
Edited by Stan Hu