Downloading a lot of small files from git-lfs takes a while in runners

Summary

We have a repo with a bunch of small images. Those images are stored in git-lfs. When we run builds, the runner has to download those images (if it doesn't already have them). It seems to download them one by one instead of all at once. This whole process takes a long time.

My hypothesis is that this is happening because gitlab-runner is not using the git-lfs batch api to download files.

Steps to reproduce

Create a repo
Setup git-lfs
Add a lot of small files and track those with git-lfs
Start a build

Actual behavior

The files are downloaded one by one and the build is slowed down because of this

Expected behavior

The files are downloaded using git-lfs' batch api which does not take that much time.

Relevant logs and/or screenshots

Running with gitlab-runner 10.5.0 (80b03db9)
  on REDACTED 814fac52
Using Docker executor with image REDACTED/docker/catalog-deploy ...
Pulling docker image REDACTED/docker/catalog-deploy ...
Using docker image sha256:8fd50ea2367a9c0e5d6ab7b2b945edd834961e50a5a5f8a63c164caab9557950 for REDACTED/docker/catalog-deploy ...
Running on runner-814fac52-project-168-concurrent-0 via REDACTED...
Cloning repository...
Cloning into '/builds/catalog/tires'...
Checking out bac84e15 as 4-export-using-media-index...
Downloading content/main/_media/altenzo/mud-basher/default.png (209 KB)
Downloading content/main/_media/altenzo/mud-basher/treadzoom.png (282 KB)
Downloading content/main/_media/altenzo/sports-comforter/default.png (188 KB)
Downloading content/main/_media/altenzo/sports-comforter/treadzoom.png (177 KB)
Downloading content/main/_media/altenzo/sports-equator/default.png (215 KB)
Downloading content/main/_media/altenzo/sports-equator/treadzoom.png (136 KB)
Downloading content/main/_media/altenzo/sports-explorer/default.png (170 KB)
Downloading content/main/_media/altenzo/sports-explorer/treadzoom.png (126 KB)
Downloading content/main/_media/altenzo/sports-navigator/default.png (248 KB)
Downloading content/main/_media/altenzo/sports-navigator/treadzoom.png (160 KB)
Downloading content/main/_media/amp/mud-terrain-attack-m-t/default.png (91 KB)
Downloading content/main/_media/amp/mud-terrain-attack-m-t/treadzoom.png (125 KB)
Downloading content/main/_media/amp/terrain-gripper-a-t/default.png (224 KB)
Downloading content/main/_media/amp/terrain-gripper-a-t/treadzoom.png (109 KB)
Downloading content/main/_media/amp/terrain-master-m-t/default.png (301 KB)
Downloading content/main/_media/amp/terrain-master-m-t/treadzoom.png (203 KB)
Downloading content/main/_media/bfgoodrich/advantage-t-a-sport-lt/default.png (990 KB)
Downloading content/main/_media/bfgoodrich/advantage-t-a-sport-lt/treadzoom.png (397 KB)
Downloading content/main/_media/bfgoodrich/advantage-t-a-sport/default.png (247 KB)
Downloading content/main/_media/bfgoodrich/advantage-t-a-sport/treadzoom.png (240 KB)
Downloading content/main/_media/bfgoodrich/advantage-t-a/default.png (198 KB)
Downloading content/main/_media/bfgoodrich/advantage-t-a/treadzoom.png (113 KB)
Downloading content/main/_media/bfgoodrich/all-terrain-t-a-ko2---ko/default.png (381 KB)
Downloading content/main/_media/bfgoodrich/all-terrain-t-a-ko2---ko/treadzoom.png (370 KB)
Downloading content/main/_media/bfgoodrich/commercial-t-a-all-season-2/default.png (102 KB)
Downloading content/main/_media/bfgoodrich/commercial-t-a-all-season-2/treadzoom.png (40 KB)
Downloading content/main/_media/bfgoodrich/commercial-t-a-traction-studded-clout%C3%A9/default.png (200 KB)
Downloading content/main/_media/bfgoodrich/commercial-t-a-traction-studded-clout%C3%A9/treadzoom.png (275 KB)
Downloading content/main/_media/bfgoodrich/commercial-t-a-traction/default.png (190 KB)
Downloading content/main/_media/bfgoodrich/commercial-t-a-traction/treadzoom.png (111 KB)
Downloading content/main/_media/bfgoodrich/g-force-comp-2-a-s/default.png (219 KB)
Downloading content/main/_media/bfgoodrich/g-force-comp-2-a-s/treadzoom.png (110 KB)
Downloading content/main/_media/bfgoodrich/g-force-sport-comp-2/default.png (245 KB)
Downloading content/main/_media/bfgoodrich/g-force-sport-comp-2/treadzoom.png (87 KB)
Downloading content/main/_media/bfgoodrich/g-force-super-sport-a-s-h-v/default.png (88 KB)
... This continues for a while ...

Environment description

We are running a custom install of Gitlab CE with 4 runners that are running on ubuntu machines. The runners are all configured to build using the docker executor.

Gitlab version (omnibus)

# gitlab-rake gitlab:env:info

System information
System:		Debian 8.10
Current User:	git
Using RVM:	no
Ruby Version:	2.3.6p384
Gem Version:	2.6.13
Bundler Version:1.13.7
Rake Version:	12.3.0
Redis Version:	3.2.11
Git Version:	2.14.3
Sidekiq Version:5.0.5
Go Version:	unknown

GitLab information
Version:	10.5.2
Revision:	b951e0d
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	postgresql
URL:		https://repo.pmcti.re
HTTP Clone URL:	https://repo.pmcti.re/some-group/some-project.git
SSH Clone URL:	ssh://git@repo.pmcti.re:2022/some-group/some-project.git
Using LDAP:	no
Using Omniauth:	no

GitLab Shell
Version:	6.0.3
Repository storage paths:
- default: 	/var/opt/gitlab/git-data/repositories
Hooks:		/opt/gitlab/embedded/service/gitlab-shell/hooks
Git:		/opt/gitlab/embedded/bin/git

Docker info on runner

# docker info
Containers: 7
 Running: 1
 Paused: 0
 Stopped: 6
Images: 2
Server Version: 17.05.0-ce
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 17
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9048e5e50717ea4497b757314bad98ea3763c145
runc version: 9c2d8d184e5da67c95d601382adf14862e4f2228
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-116-generic
Operating System: Ubuntu 16.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.765GiB
Name: station-delta
ID: L2DC:7LC5:7BA7:J53X:TVXY:J6RC:CHAR:3GBU:TUSR:Q2QD:5F4U:XQAD
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

Used GitLab Runner version

# gitlab-runner --version
Version:      10.5.0
Git revision: 80b03db9
Git branch:   10-5-stable
GO version:   go1.8.5
Built:        2018-02-22T09:18:33+00:00
OS/Arch:      linux/amd64