Mirrored project with LFS files returns Object does not exist on the server or you don't have permission to access it.

Summary

Doing the same thing directly from the GitHub repo won't trigger the error.

Additionally, the mirror status appears to be complete and successful when it is actually incomplete.

Steps to reproduce

  1. Mirror https://github.com/Unity-Technologies/Graphics
  2. Clone it on to your local machine
  3. Run git lfs fetch --all

Do the same for the GitHub repo directly and no error will show up.

What is the current bug behavior?

You get an error and the lfs fetch fails.

The mirror status received from initial sync appears to be complete and successful, when it is actually incomplete.

What is the expected correct behavior?

lfs fetch to work like with the GitHub Repo.

We should show the actual status of the initial sync correctly.

Large Premium customer: https://gitlab.zendesk.com/agent/tickets/156723 (internal only)

Proposed solution

When we create a Mirror project, we trigger the RepositoryImportWorker, that calls a service that imports the repository and after that downloads the LFS objects.

To download the LFS objects, we make several calls to the source server requesting batches of 100 LFS object's URLs, and then we put all the URLs in a list. And after getting all the URLs, we start downloading them.

As mentioned in #217925 (comment 389947865), it can take a long time to get all the URLs, and because GitHub's URLs expire after 60 minutes, by the time we start downloading the files, the URLs can have expired.

So a proper solution would be to request 100 URLs, download the files, then request more 100 URLs, download the files, and repeat the process until all files are downloaded. (from @rodrigo.tomonari comment)

We should also show the actual status of the initial sync correctly (issue for this).

Possible workaround

Edited by Magdalena Frankiewicz