Introduce use_stream and send_stream methods
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
use_stream (ex. use_file)
Summary
Pulles a stream from a file in ObjectStorage, and read the content without storing it in FileStorage. We already have a similar feature HttpIO, but we need to extend the feature to make it fully-compatible with any usecases.
Why we need
Currently, we are using use_file, which stores a remote file to local FileStorage everytime. However, it'll be removed due to Cloud Native-compatible GitLab application, because we're movign away from FileStorage.
Usecases
- Artifact Browser (Fetching a metadata from remote storage to show contents of artifacts)
- PagesWorker (Fetching an artifact from remote storage to deploy to pages)
send_stream (ex. send_data)
Summary
Sends a stream to a client from ObjectStorage, FileStorage, Redis and DB, pretty much anywhere, regardless of the store.
Why we need
Currently, we are using send_data, however, we need to allocate the whole data in Unicorn memory before we pass it to send_data, so it's potentially being a target of DDoS.
Usecases
- https://gitlab.com/gitlab-org/gitlab-ee/issues/4607 (New CI Job live-trace architecture)
-
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/18101 (db raw trace needs to use
send_data)