Deprecate background upload
Problem to solve
GitLab currently supports object storage for a range of providers. It is highly configurable. There are two types of upload-to-object-storage mechanism:
- Background upload
- Direct upload
Two is too many. We should have a single mechanism if at all possible.
Target audience
- Sidney, Systems Administrator, https://design.gitlab.com/research/personas#persona-sidney
Further details
Uploads to object storage should happen through direct_upload so that GitLab deployments are cloud-native-ready.
The only known case where background upload is necessary is when using an object storage provider for which direct uploads are not working. We found this to be true for OpenStack when using their Swift API: https://docs.gitlab.com/ee/administration/object_storage.html#openstack-compatible-connection-settings. We do support Open Stack through their S3 compatible API, however.
According to https://app.periscopedata.com/app/gitlab/679200/Enablement::Memory, OpenStack is in use in self-managed installations for the following buckets:
- lfs
- artifacts
- packages
The reported events are 3 orders of magnitude less frequent than the most popular options like AWS.
Proposal
Remove background upload as an option, and always use direct upload instead
What does success look like, and how can we measure that?
As a systems administrator, I don't have to think about how GitLab orchestrates the transfer of files into object storage. There should be a single, right way to do it, that works with all supported object stores, and doesn't require lots of complicated moving parts in the background. Namely: direct upload. We should remove background upload if at all possible.