Make Runner automatically upload artifacts to Object Storage
The following discussion from !148 (merged) should be addressed:
-
@nick.thomas started a discussion: (+7 comments) I do have a problem with this approach in general, which is that it can double the amount of time an artifact upload takes, and we already have complaints about that.
The bytes must be streamed from the CI runner to gitlab, and then from gitlab to the object store, before the job is considered complete.
An alternative implementation would be to pass the upload URL all the way to the runner. It would stream the bytes directly to the object store. When the runner tells gitlab the job is completed, it could then enqueue a sidekiq job to asynchronously download the ZIP file from the store and create the metadata.
It's a big rethink of the architecture, so maybe we can consider it as a second iteration?