Workhorse direct upload to Object Storage for artifacts on GKE
Description
Artifacts coming from GitLab Runner should be saved locally on the GitLab host before they are transferred to Object Storage (in batch, asynchronously). This requires having local storage (shared disks or similar), and it's not compatible with our cloud native approach.
Proposal
Make GitLab upload artifacts directly on Object Storage, without storing them locally.
-
Don't modify the runnners: they will still upload artifacts to workhorse
-
Workhorse takes the incoming stream and uploads to it a temporarly bucket in object storage
-
Once the upload to object storage is complete, workhorse notifies ruby, providing the temporary storage location
-
Rails moves the artifact to a new non-temporary location in object storage and then updates the state of the job (and does whatever it currently does with a newly uploaded artifact)
Links / references
Requires: https://gitlab.com/gitlab-org/gitlab-ee/issues/4184
Future step will be make the runner directly upload artifacts to Object Storage, using GitLab only to coordinate the process (https://gitlab.com/gitlab-org/gitlab-ee/issues/2348).