Skip to content

Create guidelines for uploads

In workhorse we have special rules for handling uploads, instead of wasting a unicorn process on I/O operation, we perform the upload on a cheaper go-routine. This process can also handle direct upload to object storage (There is this not so small video that explain a bit of what happens in workhorse).

The problem is that each new feature may need also a workhorse MR to take advantage of this.

Our NPM package doesn't have workhorse upload acceleration, and members of the wider community are contributing new features based on that https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9012#note_179607151 increasing our ~"technical debt"

Part of the complexity of this topic is reflected in &483 but I think this is even harder because today we don't know:

  1. who owns object storage across all the product categories
  2. what are the minimum requirements for merging a new kind of upload

Also we are working on storage and limits management, but not having a unified strategy for uploads may increase the chances of something not being properly accounted for.

Current status

This paragraph was moved to &1802

Upload encodings

From the analysis, we found only 3 kinds of file encoding in our uploads.

  1. multipart: multipart/form-data is the most common and we support it
  2. body: some APIs uploads files as the whole request body. It's supported
  3. 🔴 JSON: some JSON API uploads files as base64 encoded strings. This is not supported in workhorse - gitlab-workhorse#226 (moved)

Development log

Status

Decisions

Uploading technologies

This paragraph was moved to gitlab-org/gitlab-ce!31290 inside https://gitlab.com/gitlab-org/gitlab-ce/blob/ac-uploads-docs/doc/development/uploads.md

Edited by Alessio Caiazza