Deprecate background upload

Problem to solve

GitLab currently supports object storage for a range of providers. It is highly configurable. There are two types of upload-to-object-storage mechanism:

• Background upload
• Direct upload

Two is too many. We should have a single mechanism if at all possible.

Target audience

• Sidney, Systems Administrator, https://design.gitlab.com/research/personas#persona-sidney

Further details

As far as I'm aware, both mechanisms are compatible with all the stores we support. However, direct upload is superior in that it doesn't require us to keep temporary shared storage around, do expensive work in sidekiq, or experience race conditions when accessing the file.

Proposal

Remove background upload as an option, and always use direct upload instead

What does success look like, and how can we measure that?

As a systems administrator, I don't have to think about how GitLab orchestrates the transfer of files into object storage. There should be a single, right way to do it, that works with all supported object stores, and doesn't require lots of complicated moving parts in the background. Namely: direct upload. We should remove background upload if at all possible.

Links / references