Slow git pulls

I attempted to pull via SSH with GIT_TRACE=1 on, and this is what I see:

  1. 5 seconds to run git-upload-pack
  2. 3 seconds to run rev-list

Both issues sound like they could be related to slow disk access via NFS, but I haven't actually tried to profile what goes on with git-upload-pack or rev-list yet. This may be worth doing, especially as we look to rolling out CephFS.

@chriscool, do you have any insights that might help?

$ GIT_TRACE=1 git pull gitlab-push
14:35:48.603995 git.c:350               trace: built-in: git 'pull' 'gitlab-push'
14:35:48.610031 run-command.c:336       trace: run_command: 'fetch' '--update-head-ok' 'gitlab-push'
14:35:48.611617 exec_cmd.c:120          trace: exec: 'git' 'fetch' '--update-head-ok' 'gitlab-push'
14:35:48.614578 git.c:350               trace: built-in: git 'fetch' '--update-head-ok' 'gitlab-push'
14:35:48.630744 run-command.c:336       trace: run_command: 'ssh' 'git@gitlab.com' 'git-upload-pack '\''gitlab-org/gitlab-ce.git'\'''
14:35:53.635069 run-command.c:336       trace: run_command: 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet'
remote: Counting objects: 265, done.
remote: Compressing objects: 100% (89/89), done.
14:35:56.804687 run-command.c:336       trace: run_command: 'index-pack' '--stdin' '-v' '--fix-thin' '--keep=fetch-pack 11061 on jet.local' '--pack_header=2,265'
14:35:56.805890 exec_cmd.c:120          trace: exec: 'git' 'index-pack' '--stdin' '-v' '--fix-thin' '--keep=fetch-pack 11061 on jet.local' '--pack_header=2,265'
14:35:56.809420 git.c:350               trace: built-in: git 'index-pack' '--stdin' '-v' '--fix-thin' '--keep=fetch-pack 11061 on jet.local' '--pack_header=2,265'
remote: Total 265 (delta 221), reused 208 (delta 175)
Receiving objects: 100% (265/265), 34.70 KiB | 0 bytes/s, done.
Resolving deltas: 100% (221/221), completed with 90 local objects.
14:35:56.933700 run-command.c:336       trace: run_command: 'rev-list' '--objects' '--stdin' '--not' '--all'
14:35:56.934373 exec_cmd.c:120          trace: exec: 'git' 'rev-list' '--objects' '--stdin' '--not' '--all'
14:35:56.937605 git.c:350               trace: built-in: git 'rev-list' '--objects' '--stdin' '--not' '--all'
From gitlab.com:gitlab-org/gitlab-ce
   4b0d5a9..0e5ac6f  8-11-stable -> gitlab-push/8-11-stable
 + f6f8275...882dae6 ace-ventura -> gitlab-push/ace-ventura  (forced update)
 * [new branch]      changelog-update -> gitlab-push/changelog-update
   e88eab6..c64e977  diff-line-comment-vuejs -> gitlab-push/diff-line-comment-vuejs
   3666f69..30091ed  master     -> gitlab-push/master
14:35:57.016571 run-command.c:964       run_processes_parallel: preparing to run up to 1 tasks
14:35:57.016691 run-command.c:996       run_processes_parallel: done
14:35:57.017622 run-command.c:336       trace: run_command: 'gc' '--auto'
14:35:57.018381 exec_cmd.c:120          trace: exec: 'git' 'gc' '--auto'
14:35:57.021289 git.c:350               trace: built-in: git 'gc' '--auto'