Direct Object Storage Setup Work
Requires https://gitlab.com/gitlab-org/gitlab-ee/issues/4162
Currently ~"EE Premium" supports Object Storage, but still requires a shared file system, which can be accessed from all worker nodes in order to operate.
This is because objects are moved to object storage asynchronously, at some point after the file has been uploaded to the GitLab instance. Between the period of the file being uploaded and the file being successfully moved to the Object Storage, any accesses of that file need to be through a shared file system.
In order to make GitLab ~"Cloud Native", we need to move from this method of "indirect object storage" to "direct object storage", in which files are uploaded either directly from the client into Object Storage, or streamed via a server but directly into object storage.
The file is never written to a volume that is shared with other workers.
Objective
There are multiple deliverables that need to completed for Direct Object Storage functionality to work correctly, including https://gitlab.com/gitlab-org/gitlab-ee/issues/3507, https://gitlab.com/gitlab-org/gitlab-ee/issues/3346, https://gitlab.com/gitlab-org/gitlab-ce/issues/34317, https://gitlab.com/gitlab-org/gitlab-ee/issues/2348.
Although this work is spread across multiple engineering areas, it will likely share common functionality.
The focus of this task is to investigate and deliver any shared functionality that needs to be in place before other teams focus on delivering Direct Object Storage functionality in their own areas.
This may include, but is not limited to:
- Decide whether GitLab EEP will continue to support the existing "indirect" object storage mechanism via Sidekiq jobs, or whether we will aim to only support direct object storage in future.
- Depending on the outcome of https://gitlab.com/gitlab-org/gitlab-ee/issues/4162, if we find CarrierWave's S3 provider is not compatible with Google Cloud Storage, add additional configuration options to GitLab to use either the S3 or Google Cloud Storage provider.
- Developing any abstractions around CarrierWave that we need to have in place before implementing Direct Object Storage features in LFS, Artifacts and Attachments.