Git over HTTP LFS requests should generate an authorization header token
The Git LFS Deep Dive notes that we should be generating an authorization header for authenticating the download of LFS objects. However, a customer noticed their LDAP server was being hammered with bind queries. As we dug in we began to suspect that LFS object downloads were repeatedly authenticating.
I confirmed this in my local environment with GDK. I added some debug logging to GitLab::Auth
at line https://gitlab.com/gitlab-org/gitlab-ce/blob/0aca6d56ca1b948e6dc4f9f29ba982a16a015438/lib/gitlab/auth.rb#L91. The output would look something like Authenticating john via #<Gitlab::Auth::Database::Authentication:0x00007ff07e5fb330>
.
The result was that there were numerous authentications when cloning a small repository with 6 LFS objects:
Authenticating john via #<Gitlab::Auth::Database::Authentication:0x00007ff07e5fb330>
Authenticating john via #<Gitlab::Auth::LDAP::Authentication:0x00007ff07e5e6ca0>
Authenticating john via #<Gitlab::Auth::Database::Authentication:0x00007ff082a3e538>
Authenticating john via #<Gitlab::Auth::LDAP::Authentication:0x00007ff082a0f670>
Authenticating john via #<Gitlab::Auth::Database::Authentication:0x00007ff088d36668>
Authenticating john via #<Gitlab::Auth::LDAP::Authentication:0x00007ff088bf6b18>
Authenticating john via #<Gitlab::Auth::Database::Authentication:0x00007ff088812e20>
Authenticating john via #<Gitlab::Auth::LDAP::Authentication:0x00007ff0888076d8>
Authenticating john via #<Gitlab::Auth::Database::Authentication:0x00007ff0900f5928>
Authenticating john via #<Gitlab::Auth::Database::Authentication:0x00007ff0822fc180>
Authenticating john via #<Gitlab::Auth::LDAP::Authentication:0x00007ff082b49e28>
Authenticating john via #<Gitlab::Auth::LDAP::Authentication:0x00007ff080c6b358>
Authenticating john via #<Gitlab::Auth::Database::Authentication:0x00007ff097af1218>
Authenticating john via #<Gitlab::Auth::Database::Authentication:0x00007ff07e4485b0>
Authenticating john via #<Gitlab::Auth::LDAP::Authentication:0x00007ff07b826a18>
Authenticating john via #<Gitlab::Auth::LDAP::Authentication:0x00007ff07e42b280>
Authenticating john via #<Gitlab::Auth::Database::Authentication:0x00007ff087fe38d0>
Authenticating john via #<Gitlab::Auth::LDAP::Authentication:0x00007ff087847748>
The Git trace output for the same clone is:
10:32 $ GIT_TRACE=1 git clone http://john@localhost:3001/root/test-lfs.git test-lfs3
10:41:11.569733 git.c:415 trace: built-in: git clone http://john@localhost:3001/root/test-lfs.git test-lfs3
Cloning into 'test-lfs3'...
10:41:11.576886 run-command.c:637 trace: run_command: git-remote-http origin http://john@localhost:3001/root/test-lfs.git
10:41:11.893372 run-command.c:637 trace: run_command: 'git credential-osxkeychain get'
10:41:11.901380 git.c:659 trace: exec: git-credential-osxkeychain get
10:41:11.901909 run-command.c:637 trace: run_command: git-credential-osxkeychain get
10:41:14.478314 run-command.c:637 trace: run_command: 'git credential-osxkeychain store'
10:41:14.485532 git.c:659 trace: exec: git-credential-osxkeychain store
10:41:14.485992 run-command.c:637 trace: run_command: git-credential-osxkeychain store
10:41:14.521592 run-command.c:637 trace: run_command: git fetch-pack --stateless-rpc --stdin --lock-pack --thin --check-self-contained-and-connected --cloning http://john@localhost:3001/root/test-lfs.git/
10:41:14.526407 git.c:415 trace: built-in: git fetch-pack --stateless-rpc --stdin --lock-pack --thin --check-self-contained-and-connected --cloning http://john@localhost:3001/root/test-lfs.git/
remote: Enumerating objects: 13, done.
remote: Counting objects: 100% (13/13), done.
remote: Compressing objects: 100% (12/12), done.
remote: Total 13 (delta 2), reused 0 (delta 0)
10:41:17.067367 run-command.c:637 trace: run_command: git unpack-objects --pack_header=2,13
10:41:17.072396 git.c:415 trace: built-in: git unpack-objects --pack_header=2,13
Unpacking objects: 100% (13/13), done.
10:41:17.081296 run-command.c:637 trace: run_command: git rev-list --objects --stdin --not --all --quiet '--progress=Checking connectivity'
10:41:17.086352 git.c:415 trace: built-in: git rev-list --objects --stdin --not --all --quiet '--progress=Checking connectivity'
10:41:17.094258 run-command.c:637 trace: run_command: 'git-lfs filter-process'
10:41:17.109266 trace git-lfs: exec: git 'version'
10:41:17.118641 trace git-lfs: exec: git 'config' '-l'
10:41:17.123731 trace git-lfs: Install hook: pre-push, force=false, path=/private/tmp/test-lfs3/.git/hooks/pre-push
10:41:17.123938 trace git-lfs: Install hook: post-checkout, force=false, path=/private/tmp/test-lfs3/.git/hooks/post-checkout
10:41:17.124105 trace git-lfs: Install hook: post-commit, force=false, path=/private/tmp/test-lfs3/.git/hooks/post-commit
10:41:17.124259 trace git-lfs: Install hook: post-merge, force=false, path=/private/tmp/test-lfs3/.git/hooks/post-merge
10:41:17.124390 trace git-lfs: Initialize filter-process
10:41:17.124560 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
10:41:17.129997 trace git-lfs: tq: running as batched queue, batch size of 100
10:41:17.135603 trace git-lfs: filepathfilter: rejecting "Screen Shot 2018-10-09 at 10.06.42 AM.png" via []
10:41:17.135627 trace git-lfs: filepathfilter: accepting "Screen Shot 2018-10-09 at 10.06.42 AM.png"
10:41:17.135996 trace git-lfs: filepathfilter: rejecting "Screen Shot 2019-03-14 at 1.18.01 PM.png" via []
10:41:17.136007 trace git-lfs: filepathfilter: accepting "Screen Shot 2019-03-14 at 1.18.01 PM.png"
10:41:17.136371 trace git-lfs: filepathfilter: rejecting "Screen Shot 2019-03-15 at 10.52.44 AM.png" via []
10:41:17.136381 trace git-lfs: filepathfilter: accepting "Screen Shot 2019-03-15 at 10.52.44 AM.png"
10:41:17.136732 trace git-lfs: filepathfilter: rejecting "Screen Shot 2019-03-21 at 9.53.18 AM.png" via []
10:41:17.136742 trace git-lfs: filepathfilter: accepting "Screen Shot 2019-03-21 at 9.53.18 AM.png"
10:41:17.137054 trace git-lfs: filepathfilter: rejecting "Screen Shot 2019-04-11 at 12.48.17 PM.png" via []
10:41:17.137065 trace git-lfs: filepathfilter: accepting "Screen Shot 2019-04-11 at 12.48.17 PM.png"
10:41:17.137376 trace git-lfs: filepathfilter: rejecting "Screen Shot 2019-04-29 at 9.02.19 PM.png" via []
10:41:17.137386 trace git-lfs: filepathfilter: accepting "Screen Shot 2019-04-29 at 9.02.19 PM.png"
10:41:17.137536 trace git-lfs: tq: sending batch of size 6
10:41:17.137796 trace git-lfs: api: batch 6 files
10:41:17.219589 trace git-lfs: HTTP: POST http://john@localhost:3001/root/test-lfs.git/info/lfs/objects/batch
10:41:17.389505 trace git-lfs: HTTP: 401
10:41:17.389596 trace git-lfs: setting repository access to basic
10:41:17.389604 trace git-lfs: exec: git 'config' '--replace-all' 'lfs.http://localhost:3001/root/test-lfs.git/info/lfs.access' 'basic'
10:41:17.395174 trace git-lfs: api: http response indicates "basic" authentication. Resubmitting...
10:41:17.395273 trace git-lfs: creds: git credential fill ("http", "localhost:3001", "")
10:41:17.399530 git.c:415 trace: built-in: git credential fill
10:41:17.400215 run-command.c:637 trace: run_command: 'git credential-osxkeychain get'
10:41:17.406730 git.c:659 trace: exec: git-credential-osxkeychain get
10:41:17.407274 run-command.c:637 trace: run_command: git-credential-osxkeychain get
10:41:17.434036 trace git-lfs: Filled credentials for http://john@localhost:3001/root/test-lfs.git
10:41:17.434075 trace git-lfs: HTTP: POST http://john@localhost:3001/root/test-lfs.git/info/lfs/objects/batch
10:41:19.931640 trace git-lfs: HTTP: 200
10:41:19.931659 trace git-lfs: creds: git credential approve ("http", "localhost:3001", "")
10:41:19.936153 git.c:415 trace: built-in: git credential approve
10:41:19.936781 run-command.c:637 trace: run_command: 'git credential-osxkeychain store'
10:41:19.943858 git.c:659 trace: exec: git-credential-osxkeychain store
10:41:19.944324 run-command.c:637 trace: run_command: git-credential-osxkeychain store
10:41:19.978276 trace git-lfs: HTTP: {"objects":[{"oid":"140a047e9cd3e91b32676e131a2a8ba762a1b6c163d4e1b83f8453b3da374555","size":840718,"actions":{"download":{"href":"http://localhost:3001/root/test-lfs.git/gitlab-lfs/objects/140a047e9cd3e91b32676e131a2a8ba762a1b6c163d4e1b83f8453b3da374555","header":{"Authorization":"Basic am9objpwYXNzd29yZA=="}}}},{"oid":"9bcf7d95051b3e096e8efd29606b1683a1c7f0aa7b136bad803a0b576d96722c","size":321356,"actions":{"download":{"href":"http://localhost:3001/root/test-lfs.git/gitlab-lfs/objects/9bcf7d95051b3e096e8
10:41:19.978320 trace git-lfs: HTTP: efd29606b1683a1c7f0aa7b136bad803a0b576d96722c","header":{"Authorization":"Basic am9objpwYXNzd29yZA=="}}}},{"oid":"b90259106fea485c9a2ef311fc24d5a67d093a0f1971f8540ab9c0328aeecb1f","size":74525,"actions":{"download":{"href":"http://localhost:3001/root/test-lfs.git/gitlab-lfs/objects/b90259106fea485c9a2ef311fc24d5a67d093a0f1971f8540ab9c0328aeecb1f","header":{"Authorization":"Basic am9objpwYXNzd29yZA=="}}}},{"oid":"6cb9391ceebd4cff2e12707e034920c2ef83be9911c0a25d2003d1a4b018acc6","size":58896,"actions":{"download":{"href":"http://localhost:3001/root/test-lfs.git/gitlab-lfs/objects/6cb9391ceebd4cff2e12707e034920c2ef83be9911c0a25d2003d1a4b018acc6","header":{"Authorization":"Basic am9objpwYXNzd29yZA=="}}}},{"oid":"542a87984acab2d4387f00b6d63dbc533cde1734f2286dce63ee293d04794164","size":49742,"actions":{"download":{"href":"http://localhost:3001/root/test-lfs.git/gitlab-lfs/objects/542a87984acab2d4387f00b6d63dbc533cde1734f2286dce63ee293d04794164","header":{"Authorization":"Basic am9objpwYXNzd29yZA=="}}}},{"oid":"f3d9
10:41:19.978466 trace git-lfs: HTTP: ed5edea66a89e7033ca1e4b9df1bdf8e5a656aeafdf3880481064cfa0285","size":16778,"actions":{"download":{"href":"http://localhost:3001/root/test-lfs.git/gitlab-lfs/objects/f3d9ed5edea66a89e7033ca1e4b9df1bdf8e5a656aeafdf3880481064cfa0285","header":{"Authorization":"Basic am9objpwYXNzd29yZA=="}}}}]}
10:41:19.978641 trace git-lfs: tq: starting transfer adapter "basic"
10:41:19.979008 trace git-lfs: HTTP: GET http://localhost:3001/root/test-lfs.git/gitlab-lfs/objects/140a047e9cd3e91b32676e131a2a8ba762a1b6c163d4e1b83f8453b3da374555
10:41:22.445444 trace git-lfs: HTTP: 200
10:41:22.445690 trace git-lfs: HTTP: GET http://localhost:3001/root/test-lfs.git/gitlab-lfs/objects/9bcf7d95051b3e096e8efd29606b1683a1c7f0aa7b136bad803a0b576d96722c
10:41:22.445858 trace git-lfs: HTTP: GET http://localhost:3001/root/test-lfs.git/gitlab-lfs/objects/b90259106fea485c9a2ef311fc24d5a67d093a0f1971f8540ab9c0328aeecb1f
10:41:22.445897 trace git-lfs: HTTP: GET http://localhost:3001/root/test-lfs.git/gitlab-lfs/objects/6cb9391ceebd4cff2e12707e034920c2ef83be9911c0a25d2003d1a4b018acc6
10:41:22.445913 trace git-lfs: HTTP: GET http://localhost:3001/root/test-lfs.git/gitlab-lfs/objects/542a87984acab2d4387f00b6d63dbc533cde1734f2286dce63ee293d04794164
10:41:22.446055 trace git-lfs: HTTP: GET http://localhost:3001/root/test-lfs.git/gitlab-lfs/objects/f3d9ed5edea66a89e7033ca1e4b9df1bdf8e5a656aeafdf3880481064cfa0285
10:41:22.450564 trace git-lfs: filepathfilter: rejecting "Screen Shot 2019-03-15 at 10.52.44 AM.png" via []
10:41:22.450580 trace git-lfs: filepathfilter: accepting "Screen Shot 2019-03-15 at 10.52.44 AM.png"
10:41:25.044367 trace git-lfs: HTTP: 200
10:41:25.045295 trace git-lfs: filepathfilter: rejecting "Screen Shot 2019-04-11 at 12.48.17 PM.png" via []
10:41:25.045306 trace git-lfs: filepathfilter: accepting "Screen Shot 2019-04-11 at 12.48.17 PM.png"
10:41:25.047362 trace git-lfs: HTTP: 200B | 6.00 KiB/s
10:41:25.048088 trace git-lfs: filepathfilter: rejecting "Screen Shot 2019-03-21 at 9.53.18 AM.png" via []
10:41:25.048101 trace git-lfs: filepathfilter: accepting "Screen Shot 2019-03-21 at 9.53.18 AM.png"
10:41:27.618504 trace git-lfs: HTTP: 200B | 6.00 KiB/s
10:41:27.619402 trace git-lfs: filepathfilter: rejecting "Screen Shot 2019-04-29 at 9.02.19 PM.png" via []
10:41:27.619449 trace git-lfs: filepathfilter: accepting "Screen Shot 2019-04-29 at 9.02.19 PM.png"
10:41:27.632659 trace git-lfs: HTTP: 200B | 23.00 KiB/s
10:41:27.633604 trace git-lfs: filepathfilter: rejecting "Screen Shot 2019-03-14 at 1.18.01 PM.png" via []
10:41:27.633614 trace git-lfs: filepathfilter: accepting "Screen Shot 2019-03-14 at 1.18.01 PM.png"
10:41:30.086076 trace git-lfs: HTTP: 200B | 23.00 KiB/s
10:41:30.087712 trace git-lfs: filepathfilter: rejecting "Screen Shot 2018-10-09 at 10.06.42 AM.png" via []
10:41:30.087723 trace git-lfs: filepathfilter: accepting "Screen Shot 2018-10-09 at 10.06.42 AM.png"
Filtering content: 100% (6/6), 1.30 MiB | 102.00 KiB/s, done.
10:41:30.089287 run-command.c:637 trace: run_command: /private/tmp/test-lfs3/.git/hooks/post-checkout 0000000000000000000000000000000000000000 d0a8790da07265f7cc9bc06ccf8242d421cae1da 1
10:41:30.096896 git.c:659 trace: exec: git-lfs post-checkout 0000000000000000000000000000000000000000 d0a8790da07265f7cc9bc06ccf8242d421cae1da 1
10:41:30.097334 run-command.c:637 trace: run_command: git-lfs post-checkout 0000000000000000000000000000000000000000 d0a8790da07265f7cc9bc06ccf8242d421cae1da 1
10:41:30.109334 trace git-lfs: exec: git 'version'
10:41:30.118753 trace git-lfs: exec: git 'config' '-l'
10:41:30.123519 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
10:41:30.128391 trace git-lfs: filepathfilter: rewrite ".git" as "**/.git/**"
10:41:30.128418 trace git-lfs: filepathfilter: rewrite "**/.git" as "**/.git"
10:41:30.128487 trace git-lfs: filepathfilter: rejecting "test-lfs3" via []
10:41:30.128499 trace git-lfs: filepathfilter: accepting "test-lfs3"
10:41:30.128633 trace git-lfs: filepathfilter: rejecting "Screen Shot 2019-04-29 at 9.02.19 PM.png" via []
10:41:30.128655 trace git-lfs: filepathfilter: accepting "Screen Shot 2019-04-29 at 9.02.19 PM.png"
10:41:30.128663 trace git-lfs: filepathfilter: rejecting "Screen Shot 2019-03-21 at 9.53.18 AM.png" via []
10:41:30.128667 trace git-lfs: filepathfilter: accepting "Screen Shot 2019-03-21 at 9.53.18 AM.png"
10:41:30.128674 trace git-lfs: filepathfilter: rejecting "Screen Shot 2018-10-09 at 10.06.42 AM.png" via []
10:41:30.128678 trace git-lfs: filepathfilter: accepting "Screen Shot 2018-10-09 at 10.06.42 AM.png"
10:41:30.128683 trace git-lfs: filepathfilter: rejecting "Screen Shot 2019-03-14 at 1.18.01 PM.png" via []
10:41:30.128689 trace git-lfs: filepathfilter: accepting "Screen Shot 2019-03-14 at 1.18.01 PM.png"
10:41:30.128692 trace git-lfs: filepathfilter: rejecting "Screen Shot 2019-04-11 at 12.48.17 PM.png" via []
10:41:30.128696 trace git-lfs: filepathfilter: accepting "Screen Shot 2019-04-11 at 12.48.17 PM.png"
10:41:30.128702 trace git-lfs: filepathfilter: rejecting ".gitattributes" via []
10:41:30.128707 trace git-lfs: filepathfilter: accepting ".gitattributes"
10:41:30.128713 trace git-lfs: filepathfilter: rejecting "Screen Shot 2019-03-15 at 10.52.44 AM.png" via []
10:41:30.128719 trace git-lfs: filepathfilter: accepting "Screen Shot 2019-03-15 at 10.52.44 AM.png"
10:41:30.128727 trace git-lfs: filepathfilter: rejecting ".git" via []
10:41:30.128733 trace git-lfs: filepathfilter: rejecting ".git" via ".git"
10:41:30.130201 run-command.c:46 trace: run_command: running exit handler for pid 39746
10:41:30.130341 trace git-lfs: filepathfilter: rewrite ".git" as "**/.git/**"
10:41:30.130360 trace git-lfs: filepathfilter: rewrite "**/.git" as "**/.git"
10:41:30.130400 trace git-lfs: filepathfilter: rejecting "tmp" via []
10:41:30.130407 trace git-lfs: filepathfilter: accepting "tmp"
No wonder the customer is having trouble with larger repositories.
As I expected, using a Personal Access Token eliminates all of this. In the meantime, I've suggested the customer move to using a PAT but they use this service user in many locations so it's taking time to update from the password to PAT.
Is this a regression? Am I misunderstanding the Git LFS Deep Dive and we don't use an authorization header here?