Add Azure object storage support

We looked at this in https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/4928#note_18851455, but due to the large number of dependencies we held off. Maybe things have gotten into a better state now.

We can consider fog-azure-rm as fog-azure has been deprecated. It includes implementation of Storage Accounts.

Changing the title of this issue to fog-azure-rm

I'm still concerned about the sheer number of dependencies that this gem pulls in: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/4928/diffs

ZD: https://gitlab.zendesk.com/agent/tickets/121571 (internal)

added devopssystems groupdistribution labels

added [deprecated] Accepting merge requests label

removed [deprecated] Accepting merge requests label

mentioned in issue #32013 (closed)

Adding another vote for this addition. I am encountering customers who are operating inside Azure and are either uninterested or unable to use a different provider for object storage, and using Minio as a self-managed alternative is less than ideal for large customers that do not want to be responsible for maintaining that part of their infrastructure or are concerned about scalability.

mentioned in issue #32529 (closed)

mentioned in issue #33002 (closed)

mentioned in issue gitlab-foss#55624 (moved)

mentioned in issue #33528 (closed)

mentioned in issue #33986 (closed)

mentioned in issue #34494 (closed)

added groupecosystem [DEPRECATED] label and removed groupdistribution + 1 deleted label

Adjusting the group to ecosystem to better fit with the request

mentioned in issue #35060 (closed)

mentioned in issue #35513 (closed)

mentioned in issue #36009 (closed)

mentioned in issue #36599 (closed)

mentioned in issue #37214 (closed)

mentioned in issue #37893 (closed)

ZD#140333 (GitLab Internal)

added auto updated potential proposal labels

mentioned in issue #39033 (closed)

mentioned in issue #103435 (closed)

mentioned in issue #121514 (closed)

mentioned in issue #193060 (closed)

mentioned in issue #195133 (closed)

mentioned in issue #196498 (closed)

mentioned in issue #197834 (closed)

mentioned in issue #199017 (closed)

mentioned in issue #201687 (closed)

mentioned in issue #204690 (closed)

added [deprecated] Accepting merge requests label

changed milestone to %Awaiting further demand

added devopscreate label and removed devopssystems label

I have gotten an increasing number of questions about this from customers who are planning to migrate their GitLab instance, or already have migrated, to Azure. Do we have any updates on what would be required to implement native Azure Blob support? With that, any sort of updated timeline to see this added to GitLab?

1,600 seat premium customer is requesting this feature, would allow them to significantly simplify their backup process. ZenDesk ticket # 157006 (internal link).

+SFDC link for reporting purposes:

https://gitlab.my.salesforce.com/0016100000ZxrqH

@deuley this would help improve not only the backup process for but could also aid in performance improvements, I believe.

@christiaanconover are there specific customers your team can identify that would benefit from native azure blob storage support?

Good call @jrreid, here are some customers I know of off the top of my head that would be interested in us supporting this:

@fzimmer do you have thoughts on this regarding backups? I'm not sure this is going to ever align well with the work that ~"group::ecosystem" is doing. Can you help me find the right home for this?

I actually believe there's an argument to be made that this is devopsenablement, specifically groupdistribution. This isnt specific to backups, it's specific to making GitLab loveable on all cloud platforms inclusive of Azure. @joshlambert thoughts?

@vkelkar is this something Alliances might be able to drive?

edit: cc @ljlane for Distribution

@jrreid Can you provide some information on the implications of not doing this? What are customers using for object storage in Azure without this?

Our documentation suggests using a MinIO Gateway which, while functional, adds complexity and a single point of failure for the storage layer. It essentially means that an Azure environment that uses Blob storage cannot be fully HA.

They're not- as a consequence, my aforementioned customer has block storage in excess of 1.7TB and finds backups to be quite a painful process to manage. It's also contributing to poor performance in general for their gitlab instance.

It's possible to deploy MinIO which provides an S3-compatible API as a façade layer in front of Azure's resource management, but this adds significant complexity.

Our reference architectures recommend the use of object storage for LFS, Uploads, Artifacts and other high contributors to disk utilization "due to better performance and availability.". Customers in azure can't (easily) make good on this recommendation.

I think Object Storage can be configured for a large number of items: https://docs.gitlab.com/ee/administration/object_storage.html#configuration-guides

This is not specific to backups. I think devopsenablement is a good home for this.

On a high-level I'd agree that we should support major object storage providers but I am not sure of the technical implementation and effort needed.

@stanhu mentioned a year ago that the fog azure plugin pulls in a large number of dependencies - not sure if that situation has changed by now.

Given Azure's place in the marketplace now, I think we should just add the gem and direct support into GitLab. I think we just need to load the fog/azurerm library (https://github.com/fog/fog-azure-rm/blob/master/lib/fog/azurerm/docs/storage.md).

Note that our direct upload support only supports S3 and Google Cloud Storage at the moment. Adding Azure support would require upgrading to CarrierWave v2, since v1.3.1 doesn't support it: https://github.com/carrierwaveuploader/carrierwave/blob/v1.3.1/lib/carrierwave/storage/fog.rb#L180.

Given Azure's place in the marketplace now, I think we should just add the gem and direct support into GitLab. I think we just need to load the fog/azurerm library

Agree @stanhu. @ayufan this isn't performance or memory related, but it's also not very clear where in GitLab this would fall from a responsibility standpoint. Do you have an idea of which group may be best suited to do this change?

Not having support for the #2 cloud provider is a pretty large gap.

@joshlambert In general, I think we need to assign object storage support to a specific group, but object storage touches so many different parts of the system (CI artifacts, LFS, upload attachments, etc.) whatever choice may be arbitrary (and that's okay, just as long as there is clear ownership). For example, adding fog/azurerm for backups might fall under the Geo purview, but then adding direct upload support would have to fall to another group (Plan, Create, etc.).

@stanhu I'm not sure any group is really set up to handle something like all of object storage, as it is a fairly significant amount of work. Personally I think we should try to prioritize the items and then find individual groups to drive forward from &483.

We can re-evaluate in FY21, but I'd vote for trying to find a way to load balance these across the feature group areas to avoid a concentration of "platform" type work in a group or set of groups.

Add Azure object storage support

Problem to solve

Target audience

Proposal

What does success look like, and how can we measure that?

Links / references

Designs

Child items ...

Activity

Add Azure object storage support

Problem to solve

Target audience

Proposal

What does success look like, and how can we measure that?

Links / references

Relates to

Activity