Ability to hide object storage redirects behind workhorse
At present, when GitLab holds an upload, artifact or LFS object in object storage and the user attempts to access it, they go to a URL like, e.g.:
https://gitlab.com/gitlab-org/gitlab-ce/uploads/<secret>/<filename>
This endpoint checks the user's credentials and issues a redirect to a signed URL if the user has access. This has a fixed validity period of ~10 minutes, per http://s3.amazonaws.com/doc/s3-developer-guide/RESTAuthentication.html
The user's browser then makes a request with the signed URL to the object storage server to get the upload.
Objections:
- Leaking implementation details to the user's browser
- Leaking credentials to the user's browser
- Object storage server must be accessible to the open Internet
- Where the user's browser is far away from object storage server and gitlab install, this increases page load time
Instead, we could have GitLab issue a response that is intercepted by workhorse in a similar way to X-Sendfile
. It would contact the object store and stream the bytes directly to the user in response to their original request.
Advantages:
- User's browser never sees the object store, or credentials for it
- Credential management is much easier
- Object store can be located on a private network
- Decreased page load time if workhorse is closer to the object store than the user's browser
Disadvantages:
- Increased application load on workhorse
- Increased bandwidth in cases where the object store is not gitlab-local
My preference would be to make this the only means of accessing files in object storage, but if the disadvantages are compelling, we could make this a configurable option. I think the advantages are definitely having it as an option, at least.
/cc @ayufan @DouweM @smcgivern @jacobvosmaer-gitlab
What do we want this to look like on GitLab.com ? /cc @jarv