Skip to content

Add Azure Blob Storage support

Stan Hu requested to merge sh-add-azure-blob-store into master

This merge request introduces a client for Azure Blob Storage in Workhorse. Currently customers wanting to use Azure Blob Storage have to set up a Minio Gateway (https://docs.gitlab.com/charts/advanced/external-object-storage/azure-minio-gateway.html), which isn't ideal because it requires customers to maintain their own proxy server for Azure. We have a number of customers who want native support for Azure Blob Storage.

Unlike AWS and Google, Azure needs to use an Azure client inside Workhorse to support direct uploads. Using standard HTTP transfers with pre-signed URLs with the Azure Put Blob API (https://docs.microsoft.com/en-us/rest/api/storageservices/put-blob) doesn't work because Azure doesn't support chunked transfer encoding. However, Azure does support uploading files in segments via the Put Block and Put Block List API (https://docs.microsoft.com/en-us/rest/api/storageservices/put-block), but this requires an Azure client that can speak this API.

Instead of embedding the Microsoft Azure client directly, we use the Go Cloud Development Kit (https://godoc.org/gocloud.dev/blob) to make it easier to add other object storage providers later. For example, GitLab Rails might return this JSON payload in the /internal/uploads/authorize call:

{
   "UseWorkhorseClient":true,
   "ObjectStorage":{
      "Provider":"AzureRM",
      "GoCloudConfig":{
         "URL":"azblob://test-bucket"
      }
   }
}

The azblob scheme is managed by the Go Cloud URLMux (https://godoc.org/gocloud.dev/blob#URLMux).

Converting our existing S3 client with Go Cloud should be done later (#275 (moved)).

This changes requires gitlab!38882 (merged) to work.

Omnibus configuration changes are in omnibus-gitlab!4505 (merged).

Part of gitlab#25877 (closed)

Manual testing

File type Pass?
Upload attachments ✅
LFS ✅
CI artifacts ✅
Edited by Stan Hu

Merge request reports