git lfs push keeps retrying due to 401 when uploading large files
Summary
I can't seem to upload big LFS files that take very long to upload. It just keeps on restarting and when I look at the log I always receive 401. Reading other LFS issues I understand that 401 code is normal. But I do have access to my gitlab-ce deployment and I saw that gitlab replies with 401. Yes I know that git lfs will try using no authentication first but this doesn't. It does include a correct username (I saw on the log) and get replied 401 too many times until git lfs push said too many retries. This only seem to happen for large files which take long to upload.
Steps to reproduce
Have a fresh running gitlab with LFS enabled. Also have a Nginx reverse proxy outside running with this config.
server {
listen 80;
server_name git.myhost.com;
access_log /var/log/nginx/gitlab.access.log main;
return 301 https://git.myhost.com$request_uri;
}
server {
listen 443 ssl;
server_name git.myhost.com;
ssl_certificate /etc/letsencrypt/live/git.myhost.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/git.myhost.com/privkey.pem;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers HIGH:!aNULL:!MD5;
access_log /var/log/nginx/gitlab.access.log main;
client_max_body_size 0;
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_request_buffering off;
proxy_pass http://gitlab:80;
}
}Use git for Windows My git config global
[filter "lfs"]
clean = git-lfs clean -- %f
smudge = git-lfs smudge -- %f
process = git-lfs filter-process
required = true
[credential]
helper = manager
My git config local
[core]
repositoryformatversion = 0
filemode = false
bare = false
logallrefupdates = true
ignorecase = true
[remote "origin"]
url = https://git.myhost.com/mygroup/myproject.git
fetch = +refs/heads/*:refs/remotes/origin/*
[lfs "https://git.myhost.com/mygroup/myproject.git/info/lfs"]
access = basic
locksverify = true
[branch "master"]
remote = origin
merge = refs/heads/master
[lfs "https://git.myhost.com/mygroup/myproject.git/gitlab-lfs/objects/"]
access = basic
Checkout large LFS files (maybe 1GB++, which may take up to hours to upload) then git lfs push origin master
What is the current bug behavior?
It repeats again and again with 401 Unauthorized.
What is the expected correct behavior?
It is pushed successfully in first try although will take a long time.
Relevant logs and/or screenshots
My git lfs push log with GIT_TRACE=1 and GIT_CURL_VERBOSE=1
07:51:51.909053 trace git-lfs: HTTP: PUT https://git.myhost.com/mygroup/myproject.git/gitlab-lfs/objects/860271f6f95cf47738284624ce76052a39bfd9589eb0bdb12df569f752f2a259/1322768648
> PUT /mygroup/myproject.git/gitlab-lfs/objects/860271f6f95cf47738284624ce76052a39bfd9589eb0bdb12df569f752f2a259/1322768648 HTTP/1.1
> Host: git.myhost.com
> Authorization: Basic * * * * *
> Content-Length: 1322768648
> Content-Type: application/octet-stream
> User-Agent: git-lfs/2.10.0 (GitHub; windows amd64; go 1.12.7; git a526ba6b)
>
07:51:51.909053 trace git-lfs: xfer: adapter "basic" worker 6 auth signal received
07:51:51.909053 trace git-lfs: xfer: adapter "basic" worker 3 auth signal received
07:51:51.909053 trace git-lfs: xfer: adapter "basic" worker 4 auth signal received
07:51:51.909053 trace git-lfs: xfer: adapter "basic" worker 7 auth signal received
07:51:51.909053 trace git-lfs: xfer: adapter "basic" worker 2 auth signal received
07:51:51.909053 trace git-lfs: xfer: adapter "basic" worker 1 auth signal received
07:51:51.909053 trace git-lfs: xfer: adapter "basic" worker 5 auth signal received
08:41:09.795434 trace git-lfs: HTTP: 401 GB | 97 KB/s
< HTTP/1.1 401 Unauthorized
< Content-Length: 26
< Cache-Control: no-cache
< Connection: keep-alive
< Content-Type: text/plain; charset=utf-8
< Date: Sat, 23 May 2020 01:41:10 GMT
< Referrer-Policy: strict-origin-when-cross-origin
< Server: nginx/1.16.1
< Set-Cookie: experimentation_subject_id=ImUwZGY2M2UyLWUxZDgtNDkxMS1hOWFiLTcxOTQxNGE1NjEyNSI%3D--deede383ef5628c8b519c5cf0d0af2774100e355; domain=.padmadigital.id; path=/; expires=Wed, 23 May 2040 01:41:10 -0000; secure; HttpOnly; SameSite=None
< Www-Authenticate: Basic realm="GitLab"
< X-Content-Type-Options: nosniff
< X-Download-Options: noopen
< X-Frame-Options: DENY
< X-Permitted-Cross-Domain-Policies: none
< X-Request-Id: M83NjfmXZO4
< X-Runtime: 0.100499
< X-Ua-Compatible: IE=edge
< X-Xss-Protection: 1; mode=block
<
08:41:09.795434 trace git-lfs: xfer: adapter "basic" worker 0 finished job for "860271f6f95cf47738284624ce76052a39bfd9589eb0bdb12df569f752f2a259"
08:41:09.795434 trace git-lfs: tq: retrying object 860271f6f95cf47738284624ce76052a39bfd9589eb0bdb12df569f752f2a259: Authentication required: Authorization error: https://git.myhost.com/mygroup/myproject.git/gitlab-lfs/objects/860271f6f95cf47738284624ce76052a39bfd9589eb0bdb12df569f752f2a259/1322768648
Check that you have proper access to the repository
08:41:09.795434 trace git-lfs: tq: enqueue retry #7 for "860271f6f95cf47738284624ce76052a39bfd9589eb0bdb12df569f752f2a259" (size: 1322768648)
08:41:09.795434 trace git-lfs: tq: sending batch of size 1
08:41:09.795434 trace git-lfs: api: batch 1 files
08:41:09.795434 trace git-lfs: creds: git credential cache ("https", "git.myhost.com", "")
08:41:09.795434 trace git-lfs: Filled credentials for https://git.myhost.com/mygroup/myproject.git
08:41:09.795434 trace git-lfs: HTTP: POST https://git.myhost.com/mygroup/myproject.git/info/lfs/objects/batch
> POST /mygroup/myproject.git/info/lfs/objects/batch HTTP/1.1
> Host: git.myhost.com
> Accept: application/vnd.git-lfs+json; charset=utf-8
> Authorization: Basic * * * * *
> Content-Length: 206
> Content-Type: application/vnd.git-lfs+json; charset=utf-8
> User-Agent: git-lfs/2.10.0 (GitHub; windows amd64; go 1.12.7; git a526ba6b)
>
{"operation":"upload","objects":[{"oid":"860271f6f95cf47738284624ce76052a39bfd9589eb0bdb12df569f752f2a259","size":1322768648}],"transfers":["basic","lfs-standalone-file"],"ref":{"name":"refs/heads/master"}}08:41:10.235251 trace git-lfs: HTTP: 200
< HTTP/1.1 200 OK
< Content-Length: 717
< Cache-Control: max-age=0, private, must-revalidate
< Connection: keep-alive
< Content-Type: application/json; charset=utf-8
< Date: Sat, 23 May 2020 01:41:11 GMT
< Etag: W/"ec749096f3ca87f729ddefa636340437"
< Page-Title: GitLab
< Referrer-Policy: strict-origin-when-cross-origin
< Referrer-Policy: strict-origin-when-cross-origin
< Server: nginx/1.16.1
< Set-Cookie: experimentation_subject_id=IjdkY2ViMDUzLTRjMmItNDRjNi1hYjAxLTIyYWMwOTVkNzcwNiI%3D--0de0100db8302329e84460ec9308ef773950cf69; domain=.padmadigital.id; path=/; expires=Wed, 23 May 2040 01:41:11 -0000; secure; HttpOnly; SameSite=None
< Strict-Transport-Security: max-age=31536000
< X-Content-Type-Options: nosniff
< X-Download-Options: noopen
< X-Frame-Options: DENY
< X-Permitted-Cross-Domain-Policies: none
< X-Request-Id: nUWbDMYx1Na
< X-Runtime: 0.111132
< X-Ua-Compatible: IE=edge
< X-Xss-Protection: 1; mode=block
<
08:41:10.235251 trace git-lfs: HTTP: {"objects":[{"oid":"860271f6f95cf47738284624ce76052a39bfd9589eb0bdb12df569f752f2a259","size":1322768648,"actions":{"upload":{"href":"https://git.myhost.com/mygroup/myproject.git/gitlab-lfs/objects/860271f6f95cf47738284624ce76052a39bfd9589eb0bdb12df569f752f2a259/1322768648","header":{"Authorization":"Basic c2VyZ2lvcnlhbjpleUpoYkdjaU9pSklVekkxTmlJc0luUjVjQ0k2SWtwWFZDSjkuZXlKa1lYUmhJanA3SW1GamRHOXlJam9pYzJWeVoybHZjbmxoYmlKOUxDSnFkR2tpT2lKbU1qSmpZalJpWkMwM05qUXhMVFJoT0RndFlUTTVOaTFrTnpnMk5EUTFaVGhtWkdR
{"objects":[{"oid":"860271f6f95cf47738284624ce76052a39bfd9589eb0bdb12df569f752f2a259","size":1322768648,"actions":{"upload":{"href":"https://git.myhost.com/mygroup/myproject.git/gitlab-lfs/objects/860271f6f95cf47738284624ce76052a39bfd9589eb0bdb12df569f752f2a259/1322768648","header":{"Authorization":"Basic c2VyZ2lvcnlhbjpleUpoYkdjaU9pSklVekkxTmlJc0luUjVjQ0k2SWtwWFZDSjkuZXlKa1lYUmhJanA3SW1GamRHOXlJam9pYzJWeVoybHZjbmxoYmlKOUxDSnFkR2tpT2lKbU1qSmpZalJpWkMwM05qUXhMVFJoT0RndFlUTTVOaTFrTnpnMk5EUTFaVGhtWkdR08:41:10.235251 trace git-lfs: HTTP: aUxDSnBZWFFpT2pFMU9UQXhPVGd3TnpFc0ltNWlaaUk2TVRVNU1ERTVPREEyTml3aVpYaHdJam94TlRrd01UazVPRGN4ZlEuY0RKWFZkNF9CMHZaMEJYM3ZWbVlmZW9KbUxtSGxiNHZhbFBkRU4ycEZBMA==","Content-Type":"application/octet-stream"}}}}]}
aUxDSnBZWFFpT2pFMU9UQXhPVGd3TnpFc0ltNWlaaUk2TVRVNU1ERTVPREEyTml3aVpYaHdJam94TlRrd01UazVPRGN4ZlEuY0RKWFZkNF9CMHZaMEJYM3ZWbVlmZW9KbUxtSGxiNHZhbFBkRU4ycEZBMA==","Content-Type":"application/octet-stream"}}}}]}08:41:10.236246 trace git-lfs: xfer: adapter "basic" End()
08:41:10.236246 trace git-lfs: xfer: adapter "basic" worker 6 stopping
08:41:10.236246 trace git-lfs: xfer: adapter "basic" worker 2 stopping
08:41:10.236246 trace git-lfs: xfer: adapter "basic" worker 3 stopping
08:41:10.236246 trace git-lfs: xfer: adapter "basic" worker 7 stopping
08:41:10.236246 trace git-lfs: xfer: adapter "basic" worker 5 stopping
08:41:10.236246 trace git-lfs: xfer: adapter "basic" worker 0 stopping
08:41:10.236246 trace git-lfs: xfer: adapter "basic" worker 1 stopping
08:41:10.236246 trace git-lfs: xfer: adapter "basic" worker 4 stopping
08:41:10.236246 trace git-lfs: xfer: adapter "basic" stopped
08:41:10.236246 trace git-lfs: tq: starting transfer adapter "basic"
08:41:10.236246 trace git-lfs: xfer: adapter "basic" Begin() with 8 workers
08:41:10.236246 trace git-lfs: xfer: adapter "basic" worker 1 starting
08:41:10.236246 trace git-lfs: xfer: adapter "basic" started
08:41:10.236246 trace git-lfs: xfer: adapter "basic" worker 0 starting
08:41:10.236246 trace git-lfs: xfer: adapter "basic" worker 1 waiting for Auth
08:41:10.236246 trace git-lfs: xfer: adapter "basic" worker 3 starting
08:41:10.236246 trace git-lfs: xfer: adapter "basic" worker 3 waiting for Auth
08:41:10.236246 trace git-lfs: xfer: adapter "basic" worker 6 starting
08:41:10.236246 trace git-lfs: xfer: adapter "basic" worker 6 waiting for Auth
08:41:10.236246 trace git-lfs: xfer: adapter "basic" worker 2 starting
08:41:10.236246 trace git-lfs: xfer: adapter "basic" worker 2 waiting for Auth
08:41:10.236246 trace git-lfs: xfer: adapter "basic" worker 5 starting
08:41:10.236246 trace git-lfs: xfer: adapter "basic" worker 5 waiting for Auth
08:41:10.236246 trace git-lfs: xfer: adapter "basic" worker 7 starting
08:41:10.236246 trace git-lfs: xfer: adapter "basic" worker 4 starting
08:41:10.236246 trace git-lfs: xfer: adapter "basic" worker 4 waiting for Auth
08:41:10.236246 trace git-lfs: xfer: adapter "basic" worker 7 waiting for Auth
08:41:10.236246 trace git-lfs: xfer: adapter "basic" worker 0 processing job for "860271f6f95cf47738284624ce76052a39bfd9589eb0bdb12df569f752f2a259"
08:41:10.237235 trace git-lfs: HTTP: PUT https://git.myhost.com/mygroup/myproject.git/gitlab-lfs/objects/860271f6f95cf47738284624ce76052a39bfd9589eb0bdb12df569f752f2a259/1322768648
> PUT /mygroup/myproject.git/gitlab-lfs/objects/860271f6f95cf47738284624ce76052a39bfd9589eb0bdb12df569f752f2a259/1322768648 HTTP/1.1
> Host: git.myhost.com
> Authorization: Basic * * * * *
> Content-Length: 1322768648
> Content-Type: application/octet-stream
> User-Agent: git-lfs/2.10.0 (GitHub; windows amd64; go 1.12.7; git a526ba6b)
>
08:41:10.237235 trace git-lfs: xfer: adapter "basic" worker 6 auth signal received
08:41:10.237235 trace git-lfs: xfer: adapter "basic" worker 1 auth signal received
08:41:10.237235 trace git-lfs: xfer: adapter "basic" worker 5 auth signal received
08:41:10.237235 trace git-lfs: xfer: adapter "basic" worker 2 auth signal received
08:41:10.237235 trace git-lfs: xfer: adapter "basic" worker 4 auth signal received
08:41:10.237235 trace git-lfs: xfer: adapter "basic" worker 3 auth signal received
08:41:10.237235 trace git-lfs: xfer: adapter "basic" worker 7 auth signal received
09:30:12.205952 trace git-lfs: HTTP: 401 GB | 90 KB/s
< HTTP/1.1 401 Unauthorized
< Content-Length: 26
< Cache-Control: no-cache
< Connection: keep-alive
< Content-Type: text/plain; charset=utf-8
< Date: Sat, 23 May 2020 02:30:13 GMT
< Referrer-Policy: strict-origin-when-cross-origin
< Server: nginx/1.16.1
< Set-Cookie: experimentation_subject_id=ImUxZDYwMjNiLTZhZTEtNGRjMy1hYjdkLTkzNzg3OWU1ZDcxYyI%3D--fa7c64c8fef157f7f4db650c0e44f1baf89248d4; domain=.padmadigital.id; path=/; expires=Wed, 23 May 2040 02:30:13 -0000; secure; HttpOnly; SameSite=None
< Www-Authenticate: Basic realm="GitLab"
< X-Content-Type-Options: nosniff
< X-Download-Options: noopen
< X-Frame-Options: DENY
< X-Permitted-Cross-Domain-Policies: none
< X-Request-Id: Aqu1BSxL0X1
< X-Runtime: 0.104740
< X-Ua-Compatible: IE=edge
< X-Xss-Protection: 1; mode=block
<
09:30:12.205952 trace git-lfs: xfer: adapter "basic" worker 0 finished job for "860271f6f95cf47738284624ce76052a39bfd9589eb0bdb12df569f752f2a259"
09:30:12.205952 trace git-lfs: tq: retrying object 860271f6f95cf47738284624ce76052a39bfd9589eb0bdb12df569f752f2a259: Authentication required: Authorization error: https://git.myhost.com/mygroup/myproject.git/gitlab-lfs/objects/860271f6f95cf47738284624ce76052a39bfd9589eb0bdb12df569f752f2a259/1322768648
Check that you have proper access to the repository
09:30:12.205952 trace git-lfs: tq: enqueue retry #8 for "860271f6f95cf47738284624ce76052a39bfd9589eb0bdb12df569f752f2a259" (size: 1322768648)
09:30:12.205952 trace git-lfs: tq: sending batch of size 1
09:30:12.205952 trace git-lfs: api: batch 1 files
09:30:12.205952 trace git-lfs: creds: git credential cache ("https", "git.myhost.com", "")
09:30:12.205952 trace git-lfs: Filled credentials for https://git.myhost.com/mygroup/myproject.git
09:30:12.205952 trace git-lfs: HTTP: POST https://git.myhost.com/mygroup/myproject.git/info/lfs/objects/batch
> POST /mygroup/myproject.git/info/lfs/objects/batch HTTP/1.1
> Host: git.myhost.com
> Accept: application/vnd.git-lfs+json; charset=utf-8
> Authorization: Basic * * * * *
> Content-Length: 206
> Content-Type: application/vnd.git-lfs+json; charset=utf-8
> User-Agent: git-lfs/2.10.0 (GitHub; windows amd64; go 1.12.7; git a526ba6b)
>
{"operation":"upload","objects":[{"oid":"860271f6f95cf47738284624ce76052a39bfd9589eb0bdb12df569f752f2a259","size":1322768648}],"transfers":["basic","lfs-standalone-file"],"ref":{"name":"refs/heads/master"}}09:30:12.440819 trace git-lfs: HTTP: 200
< HTTP/1.1 200 OK
< Content-Length: 717
< Cache-Control: max-age=0, private, must-revalidate
< Connection: keep-alive
< Content-Type: application/json; charset=utf-8
< Date: Sat, 23 May 2020 02:30:13 GMT
< Etag: W/"ef79c02361abce861ea7af7f4e4afff8"
< Page-Title: GitLab
< Referrer-Policy: strict-origin-when-cross-origin
< Referrer-Policy: strict-origin-when-cross-origin
< Server: nginx/1.16.1
< Set-Cookie: experimentation_subject_id=IjM0NTgzNDY5LTcwYTQtNGZlNC1iOWZkLWVlZTUzMDU0YWYxOCI%3D--66d07ab46e01e55c3037a84c0b1928eec129d9b5; domain=.padmadigital.id; path=/; expires=Wed, 23 May 2040 02:30:13 -0000; secure; HttpOnly; SameSite=None
< Strict-Transport-Security: max-age=31536000
< X-Content-Type-Options: nosniff
< X-Download-Options: noopen
< X-Frame-Options: DENY
< X-Permitted-Cross-Domain-Policies: none
< X-Request-Id: hXdurkkwwAa
< X-Runtime: 0.090486
< X-Ua-Compatible: IE=edge
< X-Xss-Protection: 1; mode=block
<
09:30:12.440819 trace git-lfs: HTTP: {"objects":[{"oid":"860271f6f95cf47738284624ce76052a39bfd9589eb0bdb12df569f752f2a259","size":1322768648,"actions":{"upload":{"href":"https://git.myhost.com/mygroup/myproject.git/gitlab-lfs/objects/860271f6f95cf47738284624ce76052a39bfd9589eb0bdb12df569f752f2a259/1322768648","header":{"Authorization":"Basic c2VyZ2lvcnlhbjpleUpoYkdjaU9pSklVekkxTmlJc0luUjVjQ0k2SWtwWFZDSjkuZXlKa1lYUmhJanA3SW1GamRHOXlJam9pYzJWeVoybHZjbmxoYmlKOUxDSnFkR2tpT2lJeU1ERXdaRFF3WXkxbVptUTVMVFJpTkdVdE9XVXlOQzFrTUdZMk1UWmpNV0l5TnpN
{"objects":[{"oid":"860271f6f95cf47738284624ce76052a39bfd9589eb0bdb12df569f752f2a259","size":1322768648,"actions":{"upload":{"href":"https://git.myhost.com/mygroup/myproject.git/gitlab-lfs/objects/860271f6f95cf47738284624ce76052a39bfd9589eb0bdb12df569f752f2a259/1322768648","header":{"Authorization":"Basic c2VyZ2lvcnlhbjpleUpoYkdjaU9pSklVekkxTmlJc0luUjVjQ0k2SWtwWFZDSjkuZXlKa1lYUmhJanA3SW1GamRHOXlJam9pYzJWeVoybHZjbmxoYmlKOUxDSnFkR2tpT2lJeU1ERXdaRFF3WXkxbVptUTVMVFJpTkdVdE9XVXlOQzFrTUdZMk1UWmpNV0l5TnpN09:30:12.440819 trace git-lfs: HTTP: aUxDSnBZWFFpT2pFMU9UQXlNREV3TVRNc0ltNWlaaUk2TVRVNU1ESXdNVEF3T0N3aVpYaHdJam94TlRrd01qQXlPREV6ZlEuekhrWWJSWEpOR3E1Z3pjdlVRcVIyTG5LV2Q1Q2MyTk5pd3JBN0stQmgzQQ==","Content-Type":"application/octet-stream"}}}}]}
aUxDSnBZWFFpT2pFMU9UQXlNREV3TVRNc0ltNWlaaUk2TVRVNU1ESXdNVEF3T0N3aVpYaHdJam94TlRrd01qQXlPREV6ZlEuekhrWWJSWEpOR3E1Z3pjdlVRcVIyTG5LV2Q1Q2MyTk5pd3JBN0stQmgzQQ==","Content-Type":"application/octet-stream"}}}}]}09:30:12.440819 trace git-lfs: xfer: adapter "basic" End()
09:30:12.440819 trace git-lfs: xfer: adapter "basic" worker 6 stopping
09:30:12.440819 trace git-lfs: xfer: adapter "basic" worker 4 stopping
09:30:12.440819 trace git-lfs: xfer: adapter "basic" worker 5 stopping
09:30:12.440819 trace git-lfs: xfer: adapter "basic" worker 2 stopping
09:30:12.440819 trace git-lfs: xfer: adapter "basic" worker 0 stopping
09:30:12.440819 trace git-lfs: xfer: adapter "basic" worker 3 stopping
09:30:12.440819 trace git-lfs: xfer: adapter "basic" worker 7 stopping
09:30:12.440819 trace git-lfs: xfer: adapter "basic" worker 1 stopping
09:30:12.441802 trace git-lfs: xfer: adapter "basic" stopped
09:30:12.441802 trace git-lfs: tq: starting transfer adapter "basic"
09:30:12.441802 trace git-lfs: xfer: adapter "basic" Begin() with 8 workers
09:30:12.441802 trace git-lfs: xfer: adapter "basic" started
09:30:12.441802 trace git-lfs: xfer: adapter "basic" worker 3 starting
09:30:12.441802 trace git-lfs: xfer: adapter "basic" worker 0 starting
09:30:12.441802 trace git-lfs: xfer: adapter "basic" worker 1 starting
09:30:12.441802 trace git-lfs: xfer: adapter "basic" worker 6 starting
09:30:12.441802 trace git-lfs: xfer: adapter "basic" worker 3 waiting for Auth
09:30:12.441802 trace git-lfs: xfer: adapter "basic" worker 6 waiting for Auth
09:30:12.441802 trace git-lfs: xfer: adapter "basic" worker 1 waiting for Auth
09:30:12.441802 trace git-lfs: xfer: adapter "basic" worker 5 starting
09:30:12.441802 trace git-lfs: xfer: adapter "basic" worker 4 starting
09:30:12.441802 trace git-lfs: xfer: adapter "basic" worker 4 waiting for Auth
09:30:12.441802 trace git-lfs: xfer: adapter "basic" worker 7 starting
09:30:12.441802 trace git-lfs: xfer: adapter "basic" worker 7 waiting for Auth
09:30:12.441802 trace git-lfs: xfer: adapter "basic" worker 5 waiting for Auth
09:30:12.441802 trace git-lfs: xfer: adapter "basic" worker 2 starting
09:30:12.441802 trace git-lfs: xfer: adapter "basic" worker 2 waiting for Auth
09:30:12.441802 trace git-lfs: xfer: adapter "basic" worker 0 processing job for "860271f6f95cf47738284624ce76052a39bfd9589eb0bdb12df569f752f2a259"
09:30:12.441802 trace git-lfs: HTTP: PUT https://git.myhost.com/mygroup/myproject.git/gitlab-lfs/objects/860271f6f95cf47738284624ce76052a39bfd9589eb0bdb12df569f752f2a259/1322768648
> PUT /mygroup/myproject.git/gitlab-lfs/objects/860271f6f95cf47738284624ce76052a39bfd9589eb0bdb12df569f752f2a259/1322768648 HTTP/1.1
> Host: git.myhost.com
> Authorization: Basic * * * * *
> Content-Length: 1322768648
> Content-Type: application/octet-stream
> User-Agent: git-lfs/2.10.0 (GitHub; windows amd64; go 1.12.7; git a526ba6b)
>
09:30:12.441802 trace git-lfs: xfer: adapter "basic" worker 1 auth signal received
09:30:12.441802 trace git-lfs: xfer: adapter "basic" worker 4 auth signal received
09:30:12.441802 trace git-lfs: xfer: adapter "basic" worker 2 auth signal received
09:30:12.441802 trace git-lfs: xfer: adapter "basic" worker 3 auth signal received
09:30:12.441802 trace git-lfs: xfer: adapter "basic" worker 7 auth signal received
09:30:12.441802 trace git-lfs: xfer: adapter "basic" worker 6 auth signal received
09:30:12.441802 trace git-lfs: xfer: adapter "basic" worker 5 auth signal received
10:18:57.996997 trace git-lfs: HTTP: 401 GB | 80 KB/s
< HTTP/1.1 401 Unauthorized
< Content-Length: 26
< Cache-Control: no-cache
< Connection: keep-alive
< Content-Type: text/plain; charset=utf-8
< Date: Sat, 23 May 2020 03:18:59 GMT
< Referrer-Policy: strict-origin-when-cross-origin
< Server: nginx/1.16.1
< Set-Cookie: experimentation_subject_id=IjVmNDQxMzVjLTI0M2YtNDYwYi1hMzA5LWY5MmQ1MmQ4NWYzNCI%3D--7dccc35475a94eafcfe78558d4d35c8be203d9e0; domain=.padmadigital.id; path=/; expires=Wed, 23 May 2040 03:18:59 -0000; secure; HttpOnly; SameSite=None
< Www-Authenticate: Basic realm="GitLab"
< X-Content-Type-Options: nosniff
< X-Download-Options: noopen
< X-Frame-Options: DENY
< X-Permitted-Cross-Domain-Policies: none
< X-Request-Id: l0NlSC08fx8
< X-Runtime: 0.092235
< X-Ua-Compatible: IE=edge
< X-Xss-Protection: 1; mode=block
<
10:18:57.996997 trace git-lfs: xfer: adapter "basic" worker 0 finished job for "860271f6f95cf47738284624ce76052a39bfd9589eb0bdb12df569f752f2a259"
10:18:57.996997 trace git-lfs: tq: refusing to retry "860271f6f95cf47738284624ce76052a39bfd9589eb0bdb12df569f752f2a259", too many retries (8)
10:18:57.996997 trace git-lfs: tq: refusing to retry "860271f6f95cf47738284624ce76052a39bfd9589eb0bdb12df569f752f2a259", too many retries (8)
10:18:57.996997 trace git-lfs: xfer: adapter "basic" End()
10:18:57.996997 trace git-lfs: xfer: adapter "basic" worker 1 stopping
10:18:57.996997 trace git-lfs: xfer: adapter "basic" worker 3 stopping
10:18:57.996997 trace git-lfs: xfer: adapter "basic" worker 2 stopping
10:18:57.996997 trace git-lfs: xfer: adapter "basic" worker 7 stopping
10:18:57.996997 trace git-lfs: xfer: adapter "basic" worker 0 stopping
10:18:57.996997 trace git-lfs: xfer: adapter "basic" worker 5 stopping
10:18:57.996997 trace git-lfs: xfer: adapter "basic" worker 4 stopping
10:18:57.996997 trace git-lfs: xfer: adapter "basic" worker 6 stopping
10:18:57.996997 trace git-lfs: xfer: adapter "basic" stopped
Uploading LFS objects: 98% (59/60), 104 MB | 80 KB/s, done.
Authentication required: Authorization error: https://git.myhost.com/mygroup/myproject.git/gitlab-lfs/objects/860271f6f95cf47738284624ce76052a39bfd9589eb0bdb12df569f752f2a259/1322768648
Check that you have proper access to the repositoryMy Nginx reverse proxy (not inside the Gitlab Docker container) reverse-proxy-access.txt
Complete logs I got from docker logs (with lots of lines that contains other projects removed) The logs may be too long, so I don't post all of it. Ask me if you need it. gitlab-docker.txt
Results of GitLab environment info
Expand for output related to GitLab environment info
gitlab-rake gitlab:env:info System information System: Current User: git Using RVM: no Ruby Version: 2.6.5p114 Gem Version: 2.7.10 Bundler Version:1.17.3 Rake Version: 12.3.3 Redis Version: 5.0.7 Git Version: 2.26.2 Sidekiq Version:5.2.7 Go Version: unknown GitLab information Version: 12.10.1 Revision: e658772bd63 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 11.7 URL: https://git.myhost.com HTTP Clone URL: https://git.myhost.com/some-group/some-project.git SSH Clone URL: ssh://git@git.myhost.com:2222/some-group/some-project.git Using LDAP: no Using Omniauth: yes Omniauth Providers: saml GitLab Shell Version: 12.2.0 Repository storage paths: - default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
gitlab-rake gitlab:check SNI
gitlab-rake gitlab:check SANITIZE=true Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 12.2.0 ? ... OK (12.2.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 6/1 ... yes 6/3 ... yes Redis version >= 4.0.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.5) Git version >= 2.22.0 ? ... yes (2.26.2) Git user has default SSH configuration? ... yes Active users: ... 5 Is authorized keys file accessible? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
Possible fixes
I don't know about the possible fixes, but I read somewhere about expires_in, which was implemented back in 11.9? So it should has fixed this issue right? I don't know if this is authentication expired issue or not.
Edited by Sergio Ryan