Skip to content

Refactor uploaders to use different upload strategies

Stan Hu requested to merge sh-refactor-uploaders into master

Previously it was particularly tricky to add a new object storage method because you had to be aware of how to deal with different Goroutines and contexts to handle the Workhorse upload flow (https://docs.gitlab.com/ee/development/uploads.html#direct-upload). In addition, the execution engine to handle this was duplicated across multiple files. The execution engine essentially did the following:

  1. Set up an upload context with a deadline
  2. Record upload metrics
  3. Initialize cleanup functions
  4. Initiate upload
  5. Validate upload ETag
  6. Do cleanup (e.g. delete the temporary file)

To reduce code duplication and to make it easier to add new object stores, the common execution sequence is now encapsulated in the uploader Execute() method. We also introduce an UploadStrategy interface that handles the details of the uploads, and Execute() calls methods on this interface.

Now adding a new object storage type is a matter of implementing the UploadStrategy interface without duplicating the execution engine.

Edited by Stan Hu

Merge request reports