Skip to content

Include LFS blobs in archives

Stan Hu requested to merge sh-active-include-lfs-in-archives into master

This commit adds the include_lfs_blobs_in_archive feature flag. When switched on, include_lfs_blobs_in_archive will enable the inclusion of LFS blobs in archive files by enabling a flag in the GetArchiveRequest RPC message. Workhorse will decode the GetArchiveRequest and send it to Gitaly, which will run a custom LFS smudge filter (added in gitaly!2577 (merged), activated via gitaly!2621 (merged)). Support for this archive flag has already been merged via !44159 (merged).

To prevent stale archive files from being served from the cache, we introduce a new version prefix in the archive path. When the feature flag is enabled, the archive path will be in the format that looks like:

/path/to/project-1/:sha/@v2/archive.zip

When the feature flag is disabled, the archive path will revert back to the original form:

/path/to/project-1/:sha/archive.zip

Relates to #15079 (closed). See #15079 (comment 411741754) for the flow diagram.

Testing

  1. Build a gitaly binary with gitaly!2621 (merged).
  2. Restart gitaly.
  3. Check out a project that has a LFS files.
  4. Download the archive.zip.
  5. Verify only LFS pointers are present in the ZIP file.
  6. In the Rails activate Feature.enable(:include_lfs_blobs_in_archive).
  7. Wait a minute and then retry downloading.
  8. Verify that the LFS blob is there.

On my Omnibus instance, I see:

# ls -alR /var/opt/gitlab/gitlab-rails/shared/cache/archive
/var/opt/gitlab/gitlab-rails/shared/cache/archive:
total 12
drwx------ 3 git git  4096 Oct 14 00:28 .
drwx------ 3 git root 4096 Sep  8 22:41 ..
drwx------ 3 git git  4096 Oct 14 01:56 project-8

/var/opt/gitlab/gitlab-rails/shared/cache/archive/project-8:
total 12
drwx------ 3 git git 4096 Oct 14 01:56 .
drwx------ 3 git git 4096 Oct 14 00:28 ..
drwx------ 3 git git 4096 Oct 14 03:13 29f5141fffd034a4735699f8f2f9dbde1acdc3fe

/var/opt/gitlab/gitlab-rails/shared/cache/archive/project-8/29f5141fffd034a4735699f8f2f9dbde1acdc3fe:
total 16
drwx------ 3 git git 4096 Oct 14 03:13 .
drwx------ 3 git git 4096 Oct 14 01:56 ..
drwx------ 2 git git 4096 Oct 14 03:13 @v2
-rw------- 1 git git  633 Oct 14 03:07 lfs-test2-master.zip

/var/opt/gitlab/gitlab-rails/shared/cache/archive/project-8/29f5141fffd034a4735699f8f2f9dbde1acdc3fe/@v2:
total 124
drwx------ 2 git git   4096 Oct 14 03:13 .
drwx------ 3 git git   4096 Oct 14 03:13 ..
-rw------- 1 git git 115501 Oct 14 03:13 lfs-test2-master.zip
root@ip-172-30-0-215:/var/opt/gitlab/gitlab-rails/shared/cache/archive#
Edited by Stan Hu

Merge request reports