Adding another vote for this addition. I am encountering customers who are operating inside Azure and are either uninterested or unable to use a different provider for object storage, and using Minio as a self-managed alternative is less than ideal for large customers that do not want to be responsible for maintaining that part of their infrastructure or are concerned about scalability.
I have gotten an increasing number of questions about this from customers who are planning to migrate their GitLab instance, or already have migrated, to Azure. Do we have any updates on what would be required to implement native Azure Blob support? With that, any sort of updated timeline to see this added to GitLab?
1,600 seat premium customer is requesting this feature, would allow them to significantly simplify their backup process. ZenDesk ticket # 157006 (internal link).
@fzimmer do you have thoughts on this regarding backups? I'm not sure this is going to ever align well with the work that ~"group::ecosystem" is doing. Can you help me find the right home for this?
I actually believe there's an argument to be made that this is devopsenablement, specifically groupdistribution. This isnt specific to backups, it's specific to making GitLab loveable on all cloud platforms inclusive of Azure. @joshlambert thoughts?
@vkelkar is this something Alliances might be able to drive?
Our documentation suggests using a MinIO Gateway which, while functional, adds complexity and a single point of failure for the storage layer. It essentially means that an Azure environment that uses Blob storage cannot be fully HA.
They're not- as a consequence, my aforementioned customer has block storage in excess of 1.7TB and finds backups to be quite a painful process to manage. It's also contributing to poor performance in general for their gitlab instance.
Our reference architectures recommend the use of object storage for LFS, Uploads, Artifacts and other high contributors to disk utilization "due to better performance and availability.". Customers in azure can't (easily) make good on this recommendation.
Given Azure's place in the marketplace now, I think we should just add the gem and direct support into GitLab. I think we just need to load the fog/azurerm library
Agree @stanhu. @ayufan this isn't performance or memory related, but it's also not very clear where in GitLab this would fall from a responsibility standpoint. Do you have an idea of which group may be best suited to do this change?
Not having support for the #2 cloud provider is a pretty large gap.
@joshlambert In general, I think we need to assign object storage support to a specific group, but object storage touches so many different parts of the system (CI artifacts, LFS, upload attachments, etc.) whatever choice may be arbitrary (and that's okay, just as long as there is clear ownership). For example, adding fog/azurerm for backups might fall under the Geo purview, but then adding direct upload support would have to fall to another group (Plan, Create, etc.).
@stanhu I'm not sure any group is really set up to handle something like all of object storage, as it is a fairly significant amount of work. Personally I think we should try to prioritize the items and then find individual groups to drive forward from &483.
We can re-evaluate in FY21, but I'd vote for trying to find a way to load balance these across the feature group areas to avoid a concentration of "platform" type work in a group or set of groups.
Larissa Lanechanged title from Consider adding fog-azure gem to support Azure's object storage. to Consider adding fog-azure-rm gem to support Azure's object storage.
changed title from Consider adding fog-azure gem to support Azure's object storage. to Consider adding fog-azure-rm gem to support Azure's object storage.
Currently they're using NFS for job artifacts and the rest of their infra is Azure, so it would make sense to keep everything there. My two cents - we should strongly consider adding support for Azure object storage. This is just anecdotal but I have noticed more and more customers who use Azure as their primary cloud provider recently, so people would definitely use this feature.
I've relabelled this given (my interpretation) the consensus that devopsenablement owns this. @joshlambert could I ask for an update on the go-forward plan for this (if any) by early next week, 2020-07-15? I've got a customer call on 2020-07-16 where I'd love to provide a sitrep.
The fog-azure-rm Gem looks like it's idle and unmaintained at the moment. We can't even install the gem right now because it's still using fog-core v1.x, while we're using fog-core v2.x.
October 2019 was the last we heard from the maintainer on this:
I played with fog-azure-rm and Microsoft's Azure SDK. One thing I found interesting is that Azure does support creating pre-signed URLs the way that S3 and Google Cloud Storage (see the discussion in https://github.com/Azure/azure-storage-php/issues/170 about its limitations) via the Shared Access Signatures feature, but
fog-azure-rm does not yet support this for the upload case. However, it should be relatively easy to add this feature.
The Azure Blob PUT API (https://docs.microsoft.com/en-us/rest/api/storageservices/put-blob) requires that you send the Content-Length of the file ahead of time. Because Transfer-Encoding: chunked is not supported, Workhorse or Rails would have to save the entire file to disk before transferring it to Azure Blob Storage. Thus we'd have to make Azure use the background upload mode, which is not ideal because on a multi-node system this requires a shared mount (e.g. NFS) to hold the files. See #27331 (closed) for more details.
@ljlane@jrreid@kvogt1 Is there value to supporting Azure Blob Storage in background mode? I personally would prefer the Minio S3 Transfer Gateway over this.
This is a shame. @bjung - is there someone we could engage with at Microsoft Azure to help add our name to those waiting for this change, and potentially understand a timeline?
Ok, I did a spike test adding the Azure SDK to Workhorse, and I got Azure object storage to work in GitLab! My test case was my son's worksheet, and color by numbers never looked so colorful.
There are a lot of moving parts to shipping this in GitLab, so I can't promise this will all land in 13.3, but maybe we can get it for 13.4.
Some assorted comments:
It's nice that Azure allows you to parts of an upload before committing them to the final destination. We ought to consider doing this with Google (via resumable uploads) and S3 to solve #216442 (closed).
I had to patch fog-azure-rm and CarrierWave to make all this work. It looks to me that nobody is seriously using Azure with these gems.
I'm starting to wonder if we should just use Minio Gateway internally within GitLab so that we don't need to keep duplicating object storage clients within both Rails and Workhorse, but as #213288 (comment 327204011) discusses, this is a complicated topic.
@stanhu Thanks for spearheading the PoC! The result is encouraging. What are the next steps in your mind for the shortest path to support Azure object storage?
Thanks @stanhu! It's a great plan to support Azure object storage quickly.
Is our long term plan to keep the fork or upstream to fog-azure-rm? I'm inclined to upstreaming, but which ultimately may require us to be a maintainer. /cc @joshlambert@ljlane@mendeni
Is our long term plan to keep the fork or upstream to fog-azure-rm? I'm inclined to upstreaming, but which ultimately may require us to be a maintainer.
Upstreaming only works if there is an active maintainer, which there does not seem to be. I'm inclined to release the gem and see what happens.
I've forked the fog-azure-rm gem and have stripped down the gem to include only the functionality needed for Azure Blob Storage access: https://gitlab.com/gitlab-org/gitlab-fog-azure-rm/-/merge_requests/1. That way we don't have to bundle a lot of unnecessary Azure dependencies with GitLab. I've gotten the relevant tests to pass.
Upstreaming only works if there is an active maintainer,
And this is one of the reasons it was not previous pursued within GitLab. Supported had been effectively discontinued, and left without a maintainer.
Stripping down to the bare minimum of what we need (as seen in previous comment greatly reduces the load impact on our own developers in maintaining the functionality required and makes this a much more reasonable task for GitLab to take on.
For Workhorse, I'm going to look at GoCloud (https://github.com/google/go-cloud) because I'm very impressed that might enable us to add an S3, Google, and a Azure blob storage client in Workhorse without much fuss.
Ok, GoCloud doesn't work right now because the APIs don't quite support the streaming mode we need. Filed an upstream issue: https://github.com/google/go-cloud/issues/2835. I'm going to stick with using the Azure SDK directly. UPDATE: Never mind, it works great.
Ok, support for this is progressing quickly. I've gotten Azure blob storage support working with GoCloud, which makes it really easy to add new object storage providers. Given the number of reviews and additional testing involved, I still think this is still much more likely to land in 13.4 than in 13.3:
This is really exciting, Asside from our recent large customer I have another that will be very interested in this about to implement on Azure. Thank you for all the progress!
And as a follow up the new multi object storage configuration doesn't seem to work with MinIO, will our Azure native configuration work at launch or should we configure each object connection separately?
And as a follow up the new multi object storage configuration doesn't seem to work with MinIO, will our Azure native configuration work at launch or should we configure each object connection separately?
What issues are you running into with MinIO? As Josh said, we use MinIO extensively, both in development, test, and even production.
Stan Humarked the checklist item Workhorse: Refactor S3Object and related classes to make it easier to add other providers: gitlab-workhorse!551 (merged) as completed
marked the checklist item Workhorse: Refactor S3Object and related classes to make it easier to add other providers: gitlab-workhorse!551 (merged) as completed
@stanhu The customer that is using this (as of today it changed) has a requirement for Customer Managed Keys in Production for the Storage Accounts that house the containers. I'm not sure this is supported by Azure Gateway for MinIO. Is this something we could implement as well with this by the next release? I believe we need to have Production up by Oct 1st by contract but will follow up.
@kvogt1 I don't think we'll have support for Customer Managed Keys by next release. As I said in #226006 (comment 389631976), this complicates a lot of things because this means every request has to send up these encryption headers. That's a significant change. In a few places, our object storage support relies on using pre-signed URLs that work without additional secrets. For example, when uploading CI artifacts, the metadata extractor downloads the uploaded ZIP file via a pre-signed URL. Same story with the GitLab Runner. We'd have to find a way to pass these secrets to this process (e.g. environment variables or a config file). It's doable, but getting these secrets everywhere we need will take time. This is the first customer I've heard where it's a requirement. Most of the time default encryption with server-managed keys suffices.
@kvogt1@dtacheny - can you confirm the above? Since this issue is closed, please open a new request for anything that is required along with timelines.