Bitbucket: Error copying LFS objects on pull mirroring with SSH URL

A customer is facing an issue when he tries to pull mirror project from our BitbucketServer instance. The repository seems to be mirrored correctly (including files in LFS), but error is displayed on the "Mirror a repository" setting page:

I was able to reproduce this on my instance.

Looks like this is related to https://gitlab.com/gitlab-org/gitlab-ee/issues/10871

cc// @jramsay putting this on your radar.

ZD: https://gitlab.zendesk.com/agent/tickets/121821

Investigation (by @vyaklushin)

Errors in Kibana: https://log.gprd.gitlab.net/goto/6a5f24e0-0a91-11ee-8afc-c9851e4645c0

The error comes from HTTParty: https://github.com/jnunemaker/httparty/blob/30f2ec15917ef75934e7e7b139d9c92c22141093/lib/httparty/request.rb#L122.

We call it from LfsDownloadLinkListService.

We make a POST request to info/lfs/objects/batch with ssh:// schema that doesn't sound right.

We should reproduce it and define the correct endpoint to download LFS objects for SSH schema. https://github.com/git-lfs/git-lfs/blob/main/docs/api/batch.md#requests

Investigation #2 (by @igor.drozdov

git-lfs converts an SSH URL into HTTPS URL and we can do it as well. However, the complex part is to authenticate a request to this endpoint. Git LFS asks servers to implement git lfs-authenticate ... SSH "endpoint" to respond with Authentication JWT token that can be used then (unrelated note: we implement such endpoint in Gitlab Shell as well).

In order to fetch LFS files correctly, we need:

• Perform a command similar to ssh git@bitbucket.com git-lfs-authenticate namespace/project download.
  • I'm hesitant to do it from GitLab Rails, I wonder if the functionality similar to the one in git-lfs can be implemented in Gitaly.
  • Then we could perform a Gitaly RPC to fetch the authentication headers
• Build an HTTPS URL from the SSH URL (similar to git-lfs)
• Extend LFS client to use the JWT token