Configure LFS tmp cache directory when using network attached shared storage

Use case:

Git LFS files are kept on network attached shared storage, NFS or SMB.

Performance Impact:

Gitlab stores temporary files for Git LFS in a temporary directory located in the LFS storage path.

def cache_dir

"#{Gitlab.config.lfs.storage_path}/tmp/cache"

end

Files are then copied to the permanent location in the LFS storage path.

When using network attached storage this results in read/writes on the same storage path, the network, which is extremely slow compared to local disk.

When Git LFS operations are running I observed 50% CPU Wait as the operating system was reading/writing to/from an NFS mount. The CPU is waiting for write confirmation from the NFS server.

Potential Performance Gains:

Writing temporarily to local storage first would half the I/O operations on the network storage path and decrease CPU utilization.

Proposal:

cache_dir should be a configurable option when using network attached shared storage like NFS or SMB.

Links Zendesk: https://gitlab.zendesk.com/agent/tickets/57859