Direct upload for GraphQL
From #235659 (comment 444046321)
We have extensive documentation on handling uploads in our codebase. Over the course of the years we invested a lot of time and effort in making uploads scalable, k8s compatible, and performant.
With the design management feature, we introduced the apollo library to handle graphql uploads but this bypassed all the workhorse optimization we have in place.
We were able to uplift graphql uploads to the disk buffered upload optimization level but this is not enough because we are going back to uploading files to object storage from the rails controller.
If we want to keep uploading with graphql we should consider implementing direct upload.
The challenge I see is that workhorse cannot efficiently inspect the content of the query, so it would upload every single file to the same bucket. But GitLab is designed with feature-based buckets, so we will likely need to add a new one, and then eventually move the uploaded object to the final destination in rails.
It might be worth watching and reading the following materials:
- Speed up the monolith, speed up the monolith building a smart reverse proxy in Go where I explain a bit of workhorse history and the challenge we faced in releasing the first cloud-native installation.
- Uploads development documentation: The problem description
- Adding new features to Workhorse