LFS Uses incorrect tmp directory
Summary
Gitlab is using an incorrect temporary directory for LFS. Files are copied form lfs-objects/tmp/upload/ to /var/opt/gitlab/gitlab-rails/tmp and THEN copied back over to lfs-objects/xx/xx. When lfs is on a different file system, the upload can fail with ENOSPC in the production logs.
Gitlab version 9.2.5.
Steps to reproduce
Create a gitlab server with a small 8GB root file system and a huge 1TB file system to store lfs-objects and repo data.
Use the following relevant config for gitlab.rb
gitlab_rails['artifacts_path'] = "/mnt/storage1/artifacts"
gitlab_rails['lfs_storage_path'] = "/mnt/storage1/lfs-objects"
gitlab_rails['pages_path'] = "/mnt/storage1/pages"
gitlab_rails['backup_path'] = "/mnt/storage1/backups"
git_data_dirs({"default" => "/mnt/storage1/git-data"})Use git to push a large 10GB file to lfs.
What is the current bug behavior?
Pushing large files to LFS buffers the files in lfs-objects/tmp/upload/, but then incorrectly copies the entire 10GB file to /var/opt/gitlab/gitlab-rails/tmp, which runs the root fs out of disk space.
What is the expected correct behavior?
Pushing large files to LFS should buffer the files in lfs-objects/tmp/upload/. The file should then be moved to lfs-objects/xx/xx directory to avoid copying the 10GB file twice and running out of disk space.
Relevant logs and/or screenshots
Errno::ENOSPC (No space left on device - sendfile):
app/controllers/projects/lfs_storage_controller.rb:73:in `block in move_tmp_file_to_storage'
app/controllers/projects/lfs_storage_controller.rb:72:in `open'
app/controllers/projects/lfs_storage_controller.rb:72:in `move_tmp_file_to_storage'
app/controllers/projects/lfs_storage_controller.rb:65:in `store_file'
app/controllers/projects/lfs_storage_controller.rb:28:in `upload_finalize'
app/controllers/application_controller.rb:281:in `set_locale'
lib/gitlab/middleware/multipart.rb:93:in `call'
lib/gitlab/request_profiler/middleware.rb:14:in `call'
lib/gitlab/middleware/go.rb:16:in `call'
lib/gitlab/etag_caching/middleware.rb:10:in `call'
lib/gitlab/request_context.rb:18:in `call'Possible fixes
Try to avoid copying LFS files out of the path rooted at #{Gitlab.config.lfs.storage_path} to other directories. Move the file if possible to avoid copying gigantic files.