Very bad performance and high cpu load with Git LFS pull/clone

Summary

Very bad performance of lfs clone/fetch and high cpu load on server.

Steps to reproduce

I think this can happen if there are many smaller files tracked via git lfs. I tried with a bigger unreal engine project with multiple uasset files (binary files size from about 1 kilobyte to a few hundred megabytes). Our setup runs with an apache reverse proxy, but I don't know if that is part of the issue

What is the current bug behavior?

If I clone a project with many files of different size the transfers rate is quite unstable and ranges from 500KByte/s to 40MBytes/s in a gigabit network environment. Further the cpu load on the server increase to about 40% on a dual CPU E5-2680 v2 setup with 64gb of ram. I find this quite unacceptable for only one client doing a pull/clone!

What is the expected correct behavior?

Much lower (cpu-) load on the server. More stable (and faster) transfers.

Relevant logs and/or screenshots

I think it has to do with authentication for each transfered file. See htop and perf output. Is it really neccessary to authenticate for every file? I mean the file id is a cryptographic hash so there is basically no chance to guess them anyways. Eventually there could be an option for a trusted lan to disable it?

If you need more infos or logs I'm happy to provide them.

Results of GitLab environment info

Expand for output related to GitLab environment info

sudo gitlab-rake gitlab:env:info
System information
System:         Ubuntu 18.04
Current User:   git
Using RVM:      no
Ruby Version:   2.6.3p62
Gem Version:    2.7.9
Bundler Version:1.17.3
Rake Version:   12.3.3
Redis Version:  3.2.12
Git Version:    2.22.0
Sidekiq Version:5.2.7
Go Version:     unknown
GitLab information
Version:        12.4.3
Revision:       4d477238500
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     10.7
URL:            https://gitlab.independent-arts-software.de
HTTP Clone URL: https://gitlab.independent-arts-software.de/some-group/some-project.git
SSH Clone URL:  git@gitlab.independent-arts-software.de:some-group/some-project.git
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers:
GitLab Shell
Version:        10.2.0
Repository storage paths:

• default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check

sudo gitlab-rake gitlab:check SANITIZE=true
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 10.2.0 ? ... OK (10.2.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
2/1 ... yes
2/2 ... yes
3/5 ... yes
2/6 ... yes
2/9 ... yes
2/10 ... yes
2/11 ... yes
2/12 ... yes
2/13 ... yes
2/15 ... yes
2/16 ... yes
2/18 ... yes
2/19 ... yes
2/20 ... yes
2/28 ... yes
2/29 ... yes
2/31 ... yes
2/32 ... yes
2/34 ... yes
2/35 ... yes
2/36 ... yes
2/37 ... yes
2/39 ... yes
2/40 ... yes
2/41 ... yes
2/42 ... yes
2/43 ... yes
2/44 ... yes
2/45 ... yes
2/46 ... yes
2/47 ... yes
2/48 ... yes
2/49 ... yes
2/50 ... yes
42/51 ... yes
42/52 ... yes
2/53 ... yes
2/54 ... yes
2/56 ... yes
2/57 ... yes
36/58 ... yes
2/59 ... yes
2/60 ... yes
2/61 ... yes
2/62 ... yes
2/63 ... yes
2/64 ... yes
2/65 ... yes
2/66 ... yes
2/67 ... yes
2/68 ... yes
2/69 ... yes
2/70 ... yes
2/71 ... yes
2/72 ... yes
2/105 ... yes
2/106 ... yes
2/108 ... yes
60/109 ... yes
60/110 ... yes
60/111 ... yes
2/112 ... yes
2/113 ... yes
2/114 ... yes
2/115 ... yes
2/116 ... yes
2/117 ... yes
2/118 ... yes
2/119 ... yes
2/121 ... yes
2/122 ... yes
8/124 ... yes
2/125 ... yes
2/126 ... yes
2/127 ... yes
3/128 ... yes
2/129 ... yes
2/130 ... yes
2/131 ... yes
2/132 ... yes
2/133 ... yes
2/134 ... yes
2/135 ... yes
2/136 ... yes
2/137 ... yes
2/138 ... yes
2/139 ... yes
3/140 ... yes
3/142 ... yes
2/143 ... yes
2/144 ... yes
2/145 ... yes
2/146 ... yes
2/147 ... yes
2/148 ... yes
2/149 ... yes
2/150 ... yes
2/152 ... yes
2/153 ... yes
2/154 ... yes
2/155 ... yes
2/156 ... yes
2/157 ... yes
2/158 ... yes
3/159 ... yes
2/160 ... yes
74/161 ... yes
74/162 ... yes
74/163 ... yes
74/164 ... yes
74/165 ... yes
74/166 ... yes
74/167 ... yes
74/168 ... yes
74/169 ... yes
74/170 ... yes
74/171 ... yes
74/172 ... yes
74/173 ... yes
74/174 ... yes
74/175 ... yes
74/176 ... yes
74/177 ... yes
74/178 ... yes
74/179 ... yes
74/180 ... yes
74/181 ... yes
74/182 ... yes
74/183 ... yes
74/184 ... yes
74/185 ... yes
74/186 ... yes
74/187 ... yes
74/188 ... yes
74/189 ... yes
74/190 ... yes
2/191 ... yes
2/192 ... yes
2/193 ... yes
2/194 ... yes
2/195 ... yes
2/196 ... yes
2/197 ... yes
2/198 ... yes
2/199 ... yes
2/200 ... yes
2/201 ... yes
2/202 ... yes
46/203 ... yes
2/204 ... yes
2/207 ... yes
2/208 ... yes
2/209 ... yes
2/210 ... yes
2/211 ... yes
2/212 ... yes
2/213 ... yes
2/214 ... yes
2/215 ... yes
80/218 ... yes
80/221 ... yes
80/222 ... yes
80/223 ... yes
80/224 ... yes
80/225 ... yes
80/226 ... yes
80/227 ... yes
80/228 ... yes
80/229 ... yes
80/230 ... yes
80/231 ... yes
80/232 ... yes
80/233 ... yes
80/234 ... yes
80/235 ... yes
80/236 ... yes
80/237 ... yes
80/238 ... yes
80/239 ... yes
80/240 ... yes
80/241 ... yes
80/242 ... yes
80/243 ... yes
80/244 ... yes
80/245 ... yes
80/246 ... yes
80/247 ... yes
80/248 ... yes
80/249 ... yes
80/250 ... yes
80/251 ... yes
80/252 ... yes
80/253 ... yes
80/254 ... yes
80/255 ... yes
80/256 ... yes
80/257 ... yes
80/258 ... yes
80/259 ... yes
80/260 ... yes
80/261 ... yes
80/262 ... yes
80/263 ... yes
80/264 ... yes
80/265 ... yes
80/266 ... yes
80/267 ... yes
80/268 ... yes
80/269 ... yes
80/270 ... yes
80/271 ... yes
80/272 ... yes
80/273 ... yes
80/274 ... yes
80/275 ... yes
80/276 ... yes
80/277 ... yes
80/278 ... yes
80/279 ... yes
80/280 ... yes
80/281 ... yes
80/282 ... yes
80/283 ... yes
80/284 ... yes
80/285 ... yes
80/286 ... yes
80/287 ... yes
80/288 ... yes
80/289 ... yes
80/290 ... yes
80/291 ... yes
80/292 ... yes
80/293 ... yes
80/294 ... yes
80/295 ... yes
80/296 ... yes
80/297 ... yes
80/298 ... yes
80/299 ... yes
80/300 ... yes
80/301 ... yes
80/302 ... yes
80/303 ... yes
80/304 ... yes
80/305 ... yes
80/306 ... yes
80/307 ... yes
80/308 ... yes
80/309 ... yes
80/310 ... yes
80/311 ... yes
80/312 ... yes
80/313 ... yes
80/314 ... yes
80/315 ... yes
80/316 ... yes
80/317 ... yes
80/318 ... yes
80/319 ... yes
80/320 ... yes
80/321 ... yes
80/322 ... yes
80/323 ... yes
80/324 ... yes
80/325 ... yes
80/326 ... yes
80/327 ... yes
80/328 ... yes
80/329 ... yes
80/330 ... yes
80/331 ... yes
80/332 ... yes
80/333 ... yes
80/334 ... yes
80/335 ... yes
80/336 ... yes
80/337 ... yes
80/338 ... yes
80/339 ... yes
80/340 ... yes
80/341 ... yes
80/342 ... yes
80/343 ... yes
80/344 ... yes
80/345 ... yes
80/346 ... yes
80/347 ... yes
80/348 ... yes
80/349 ... yes
80/350 ... yes
80/351 ... yes
80/352 ... yes
80/353 ... yes
80/354 ... yes
80/355 ... yes
80/356 ... yes
80/357 ... yes
80/358 ... yes
80/359 ... yes
80/360 ... yes
80/361 ... yes
80/362 ... yes
80/363 ... yes
80/364 ... yes
80/365 ... yes
80/366 ... yes
80/367 ... yes
80/368 ... yes
80/369 ... yes
80/370 ... yes
80/371 ... yes
80/372 ... yes
80/373 ... yes
80/374 ... yes
80/375 ... yes
80/376 ... yes
80/377 ... yes
80/378 ... yes
80/379 ... yes
80/380 ... yes
80/381 ... yes
80/382 ... yes
80/383 ... yes
80/384 ... yes
80/385 ... yes
80/386 ... yes
80/387 ... yes
80/388 ... yes
80/389 ... yes
80/390 ... yes
80/391 ... yes
80/392 ... yes
80/393 ... yes
80/394 ... yes
80/395 ... yes
80/396 ... yes
80/397 ... yes
80/398 ... yes
80/399 ... yes
80/400 ... yes
80/401 ... yes
80/402 ... yes
80/403 ... yes
80/404 ... yes
80/405 ... yes
80/406 ... yes
80/407 ... yes
80/408 ... yes
80/409 ... yes
80/410 ... yes
80/411 ... yes
80/412 ... yes
80/413 ... yes
80/414 ... yes
80/415 ... yes
80/416 ... yes
80/417 ... yes
80/418 ... yes
80/419 ... yes
80/420 ... yes
80/421 ... yes
80/422 ... yes
80/423 ... yes
80/424 ... yes
80/425 ... yes
80/426 ... yes
80/427 ... yes
80/428 ... yes
80/429 ... yes
80/430 ... yes
80/431 ... yes
80/432 ... yes
80/433 ... yes
80/434 ... yes
80/435 ... yes
80/436 ... yes
80/437 ... yes
80/438 ... yes
80/439 ... yes
80/440 ... yes
80/441 ... yes
80/442 ... yes
80/443 ... yes
80/444 ... yes
80/445 ... yes
80/446 ... yes
80/447 ... yes
80/448 ... yes
80/449 ... yes
80/450 ... yes
80/451 ... yes
80/452 ... yes
80/453 ... yes
80/454 ... yes
80/455 ... yes
80/456 ... yes
80/457 ... yes
80/458 ... yes
80/459 ... yes
80/460 ... yes
80/461 ... yes
80/462 ... yes
80/463 ... yes
80/464 ... yes
80/465 ... yes
80/466 ... yes
80/467 ... yes
80/468 ... yes
80/469 ... yes
80/470 ... yes
80/471 ... yes
80/472 ... yes
80/473 ... yes
80/474 ... yes
80/475 ... yes
80/476 ... yes
80/477 ... yes
80/478 ... yes
80/479 ... yes
80/480 ... yes
80/481 ... yes
80/482 ... yes
80/483 ... yes
80/484 ... yes
80/485 ... yes
80/486 ... yes
80/487 ... yes
80/488 ... yes
80/489 ... yes
80/490 ... yes
80/491 ... yes
80/492 ... yes
80/493 ... yes
2/494 ... yes
2/495 ... yes
2/497 ... yes
2/498 ... yes
2/500 ... yes
2/501 ... yes
2/502 ... yes
4/503 ... yes
2/504 ... yes
2/505 ... yes
2/506 ... yes
2/507 ... yes
2/508 ... yes
2/509 ... yes
2/510 ... yes
2/511 ... yes
2/512 ... yes
2/513 ... yes
2/515 ... yes
2/516 ... yes
46/517 ... yes
2/519 ... yes
2/520 ... yes
2/521 ... yes
2/522 ... yes
2/523 ... yes
2/524 ... yes
2/525 ... yes
2/527 ... yes
2/528 ... yes
2/529 ... yes
2/530 ... yes
2/531 ... yes
2/532 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.3)
Git version >= 2.22.0 ? ... yes (2.22.0)
Git user has default SSH configuration? ... yes
Active users: ... 60
Is authorized keys file accessible? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished