Introduce `use_stream` and `send_stream` methods
use_stream
(ex. use_file
)
Summary
Pulles a stream from a file in ObjectStorage, and read the content without storing it in FileStorage. We already have a similar feature HttpIO
, but we need to extend the feature to make it fully-compatible with any usecases.
Why we need
Currently, we are using use_file
, which stores a remote file to local FileStorage everytime. However, it'll be removed due to Cloud Native-compatible GitLab application, because we're movign away from FileStorage.
Usecases
- Artifact Browser (Fetching a metadata from remote storage to show contents of artifacts)
- PagesWorker (Fetching an artifact from remote storage to deploy to pages)
send_stream
(ex. send_data
)
Summary
Sends a stream to a client from ObjectStorage, FileStorage, Redis and DB, pretty much anywhere, regardless of the store.
Why we need
Currently, we are using send_data
, however, we need to allocate the whole data in Unicorn memory before we pass it to send_data
, so it's potentially being a target of DDoS.
Usecases
- https://gitlab.com/gitlab-org/gitlab-ee/issues/4607 (New CI Job live-trace architecture)
-
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/18101 (db raw trace needs to use
send_data
)