Geo replication: Secondary fails to download LFS files from the primary's S3 without proxy_download
Summary
A customer set up a test Geo instance and all LFS files are failing to sync.
Other things are syncing, like job artifacts, which had proxy_download enabled.
When we enabled proxy_download for LFS, it worked.
Steps to reproduce
- Configure LFS on S3 without
proxy_download - Setup a Geo secondary
Now when the secondary attempts to sync LFS:
- It does a GET for a file against the primary
- The primary responds with a redirect to the authorized S3 URL
- Error on the secondary
What is the current bug behavior?
If the secondary stores LFS in local storage: It fails because the status code is 302, and this codepath does not follow by default.
If the secondary stores LFS in remote storage: This codepath does ::HTTP.follow.get() so it follows the redirect, but httprb adds the Gl-Geo Authorization headers from the original request to the primary, so S3 responds with 400 Bad Request saying there is more than one authorization method.
If we generate the request URL and headers, and make the request with curl, it works (200 and receives the binary data) because it follows the redirect and does not add the original Authorization headers.
What is the expected correct behavior?
The secondary should successfully download the file.
Relevant logs and/or screenshots
(Paste any relevant logs - please use code blocks (```) to format console output, logs, and code as it's tough to read otherwise.)
Output of checks
(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)
Results of GitLab environment info
GitLab 12.7.2
Possible fixes
(If you can, link to the line of code that might be responsible for the problem)