Create guidelines for uploads
In workhorse we have special rules for handling uploads, instead of wasting a unicorn process on I/O operation, we perform the upload on a cheaper go-routine. This process can also handle direct upload to object storage (There is this not so small video that explain a bit of what happens in workhorse).
The problem is that each new feature may need also a workhorse MR to take advantage of this.
Our NPM package doesn't have workhorse upload acceleration, and members of the wider community are contributing new features based on that https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9012#note_179607151 increasing our ~"technical debt"
Part of the complexity of this topic is reflected in &483 but I think this is even harder because today we don't know:
- who owns object storage across all the product categories
- what are the minimum requirements for merging a new kind of upload
Also we are working on storage and limits management, but not having a unified strategy for uploads may increase the chances of something not being properly accounted for.
Current status
This paragraph was moved to &1802
Upload encodings
From the analysis, we found only 3 kinds of file encoding in our uploads.
-
✅ multipart:multipart/form-data
is the most common and we support it -
✅ body: some APIs uploads files as the whole request body. It's supported -
🔴 JSON: some JSON API uploads files as base64 encoded strings. This is not supported in workhorse - gitlab-workhorse#226 (moved)
Development log
Status
-
Write a development guidelines about uploads !31290 (merged)
Decisions
- instead of working on the issue description, defining the problem and explaining available solution was moved to !31290 (merged)
- General discussion moved to &1802
Uploading technologies
This paragraph was moved to gitlab-org/gitlab-ce!31290 inside https://gitlab.com/gitlab-org/gitlab-ce/blob/ac-uploads-docs/doc/development/uploads.md