Skip to content
GitLab Next
  • Menu
Projects Groups Snippets
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
  • GitLab GitLab
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 43,816
    • Issues 43,816
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
    • Requirements
  • Merge requests 1,446
    • Merge requests 1,446
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
    • Test Cases
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages & Registries
    • Packages & Registries
    • Package Registry
    • Container Registry
    • Infrastructure Registry
  • Monitor
    • Monitor
    • Metrics
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Code review
    • Insights
    • Issue
    • Repository
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • GitLab.org
  • GitLabGitLab
  • Issues
  • #280819
Closed
Open
Created Nov 10, 2020 by Alessio Caiazza@nolith📦Maintainer

Direct upload for GraphQL

From #235659 (comment 444046321)

We have extensive documentation on handling uploads in our codebase. Over the course of the years we invested a lot of time and effort in making uploads scalable, k8s compatible, and performant.

With the design management feature, we introduced the apollo library to handle graphql uploads but this bypassed all the workhorse optimization we have in place.

We were able to uplift graphql uploads to the disk buffered upload optimization level but this is not enough because we are going back to uploading files to object storage from the rails controller.

If we want to keep uploading with graphql we should consider implementing direct upload.

The challenge I see is that workhorse cannot efficiently inspect the content of the query, so it would upload every single file to the same bucket. But GitLab is designed with feature-based buckets, so we will likely need to add a new one, and then eventually move the uploaded object to the final destination in rails.

It might be worth watching and reading the following materials:

  • Speed up the monolith, speed up the monolith building a smart reverse proxy in Go where I explain a bit of workhorse history and the challenge we faced in releasing the first cloud-native installation.
  • Uploads development documentation: The problem description
  • Adding new features to Workhorse
Edited Nov 10, 2020 by Alessio Caiazza
Assignee
Assign to
Time tracking