Authentication failure when doing a git clone on the secondary node if the username and password is specified in the URL and the repository is out of sync
Summary
If the repository on the secondary node is out of sync with the primary node, and you do a git clone
with the username and token in the link, for example:
$ git clone http://root:INSERT_TOKEN_HERE@10.66.181.197/root/test2
It will fail with the following error:
Cloning into 'test2'...
remote: HTTP Basic: Access denied. The provided password or token is incorrect or your account has 2FA enabled and you must use a personal access token instead of a password. See http://10.66.181.103/help/topics/git/troubleshooting_git#error-on-git-fetch-http-basic-access-denied
fatal: Authentication failed for 'http://10.66.181.197/root/test2/'
This doesn't happen if you didn't specify the authentication in the URL:
$ git clone http://10.66.181.197/root/test2
Cloning into 'test2'...
Username for 'http://10.66.181.103': root
Password for 'http://root@10.66.181.103':
warning: redirecting to http://10.66.181.103/-/push_from_secondary/2/root/test2.git/
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (3/3), 2.78 KiB | 1.39 MiB/s, done.
From checking the output of GIT_CURL_VERBOSE=2 git clone http://root:INSERT_TOKEN_HERE@10.66.181.197/root/test2
, it turns out that GitLab is removing the authentication header during the redirect.
I was able to reproduce this on GitLab 15.6.8.
Steps to reproduce
- Setup GitLab Geo.
- Temporarily stop sidekiq on the Secondary node to force some of the repositories to be out of sync.
- Run
git clone https://username:token@secondarynode/path/to/project.git/
Example Project
What is the current bug behavior?
The clone failed with an HTTP Basic: Access denied
.
What is the expected correct behavior?
The clone should succeed.
Relevant logs and/or screenshots
Here is the output when GIT_CURL_VERBOSE
is set to 2
.
# GIT_CURL_VERBOSE=2 git clone http://root:INSERT_TOKEN_HERE@10.66.181.197/root/test2
Cloning into 'test2'...
* Trying 10.66.181.197:80...
* TCP_NODELAY set
* Connected to 10.66.181.197 (10.66.181.197) port 80 (#0)
> GET /root/test2/info/refs?service=git-upload-pack HTTP/1.1
Host: 10.66.181.197
User-Agent: git/2.25.1
Accept: */*
Accept-Encoding: deflate, gzip, br
Accept-Language: C, *;q=0.9
Pragma: no-cache
* Mark bundle as not supporting multiuse
< HTTP/1.1 301 Moved Permanently
< Server: nginx
< Date: Thu, 18 May 2023 02:43:29 GMT
< Content-Type: text/html
< Content-Length: 135
< Connection: keep-alive
< Cache-Control: no-cache
< Location: http://10.66.181.197/root/test2.git/info/refs?service=git-upload-pack
< Strict-Transport-Security: max-age=63072000
< X-Request-Id: 01H0PBVC74KC76PVXVZYF9TC0H
< X-Runtime: 0.011142
< Strict-Transport-Security: max-age=63072000
< Referrer-Policy: strict-origin-when-cross-origin
<
* Ignoring the response-body
* Connection #0 to host 10.66.181.197 left intact
* Issue another request to this URL: 'http://10.66.181.197/root/test2.git/info/refs?service=git-upload-pack'
* Found bundle for host 10.66.181.197: 0xaaab07664cc0 [serially]
* Can not multiplex, even if we wanted to!
* Re-using existing connection! (#0) with host 10.66.181.197
* Connected to 10.66.181.197 (10.66.181.197) port 80 (#0)
> GET /root/test2.git/info/refs?service=git-upload-pack HTTP/1.1
Host: 10.66.181.197
User-Agent: git/2.25.1
Accept: */*
Accept-Encoding: deflate, gzip, br
Accept-Language: C, *;q=0.9
Pragma: no-cache
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Server: nginx
< Date: Thu, 18 May 2023 02:43:29 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 159
< Connection: keep-alive
< Cache-Control: no-cache
< Location: http://10.66.181.103/-/push_from_secondary/2/root/test2.git/info/refs?service=git-upload-pack
< X-Content-Type-Options: nosniff
< X-Download-Options: noopen
< X-Frame-Options: SAMEORIGIN
< X-Permitted-Cross-Domain-Policies: none
< X-Request-Id: 01H0PBVC7WTZT65F564CVMX6QM
< X-Runtime: 0.030827
< X-Xss-Protection: 1; mode=block
< Strict-Transport-Security: max-age=63072000
< Referrer-Policy: strict-origin-when-cross-origin
<
* Ignoring the response-body
* Connection #0 to host 10.66.181.197 left intact
* Issue another request to this URL: 'http://10.66.181.103/-/push_from_secondary/2/root/test2.git/info/refs?service=git-upload-pack'
* NTLM-proxy picked AND auth done set, clear picked!
* Trying 10.66.181.103:80...
* TCP_NODELAY set
* Connected to 10.66.181.103 (10.66.181.103) port 80 (#1)
> GET /-/push_from_secondary/2/root/test2.git/info/refs?service=git-upload-pack HTTP/1.1
Host: 10.66.181.103
User-Agent: git/2.25.1
Accept: */*
Accept-Encoding: deflate, gzip, br
Accept-Language: C, *;q=0.9
Pragma: no-cache
* Mark bundle as not supporting multiuse
< HTTP/1.1 401 Unauthorized
< Server: nginx
< Date: Thu, 18 May 2023 02:43:29 GMT
< Content-Type: text/plain; charset=utf-8
< Content-Length: 271
< Connection: keep-alive
< Cache-Control: no-cache
< Vary: Accept
< WWW-Authenticate: Basic realm="GitLab"
< X-Content-Type-Options: nosniff
< X-Download-Options: noopen
< X-Frame-Options: SAMEORIGIN
< X-Permitted-Cross-Domain-Policies: none
< X-Request-Id: 01H0PBVC90J4Z4QV6HW6WC9JHF
< X-Runtime: 0.020256
< X-Xss-Protection: 1; mode=block
<
* Ignoring the response-body
* Connection #1 to host 10.66.181.103 left intact
* Issue another request to this URL: 'http://10.66.181.103/-/push_from_secondary/2/root/test2.git/info/refs?service=git-upload-pack'
* Found bundle for host 10.66.181.103: 0xaaab07666c50 [serially]
* Can not multiplex, even if we wanted to!
* Re-using existing connection! (#1) with host 10.66.181.103
* Connected to 10.66.181.103 (10.66.181.103) port 80 (#1)
> GET /-/push_from_secondary/2/root/test2.git/info/refs?service=git-upload-pack HTTP/1.1
Host: 10.66.181.103
User-Agent: git/2.25.1
Accept: */*
Accept-Encoding: deflate, gzip, br
Accept-Language: C, *;q=0.9
Pragma: no-cache
* Mark bundle as not supporting multiuse
< HTTP/1.1 401 Unauthorized
< Server: nginx
< Date: Thu, 18 May 2023 02:43:29 GMT
< Content-Type: text/plain; charset=utf-8
< Content-Length: 271
< Connection: keep-alive
< Cache-Control: no-cache
< Vary: Accept
* Authentication problem. Ignoring this.
< WWW-Authenticate: Basic realm="GitLab"
< X-Content-Type-Options: nosniff
< X-Download-Options: noopen
< X-Frame-Options: SAMEORIGIN
< X-Permitted-Cross-Domain-Policies: none
< X-Request-Id: 01H0PBVC9S0YDFSPQWAKNAP7HB
< X-Runtime: 0.020512
< X-Xss-Protection: 1; mode=block
<
* Connection #1 to host 10.66.181.103 left intact
remote: HTTP Basic: Access denied. The provided password or token is incorrect or your account has 2FA enabled and you must use a personal access token instead of a password. See http://10.66.181.103/help/topics/git/troubleshooting_git#error-on-git-fetch-http-basic-access-denied
fatal: Authentication failed for 'http://10.66.181.197/root/test2/'
The authentication header is stripped out when after the redirect to the primary node.
Ideally, it should have this in the header:
Authorization: Basic AUTH
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true
)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true
)(we will only investigate if the tests are passing)