Commit 70607d82 authored by DJ Mountney's avatar DJ Mountney

Merge branch 'abubakar-azure-object-storage-docs' into 'master'

Azure External Object storage docs update

See merge request charts/gitlab!778
parents 9b12227c 1592a16d
Pipeline #59647670 (#4851) passed with stages
in 26 minutes and 26 seconds
# Azure Minio Gateway
[Minio](https://minio.io/) is an object storage server that exposes S3-compatible APIs and it has a gateway feature that allows proxying requests to Azure Blob Storage. To setup our gateway, we will make use of Azure's Web App on Linux.
To get started, make sure you have installed Azure CLI and you are logged in (`az login`). Proceed to create a [Resource group](https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-overview#resource-groups), if you don't have one already:
```
az group create --name "gitlab-azure-minio" --location "WestUS"
```
## Storage Account
Create a Storage account in your resource group, the name of the storage account must be globally unique:
```
az storage account create \
--name "gitlab-azure-minio-storage" \
--kind BlobStorage \
--sku Standard_LRS \
--access-tier Cool \
--resource-group "gitlab-azure-minio" \
--location "WestUS"
```
Retrieve the account key for the storage account:
```
az storage account show-connection-string \
--name "gitlab-azure-minio-storage" \
--resource-group "gitlab-azure-minio"
```
The output should be in the format:
```
{
"connectionString": "DefaultEndpointsProtocol=https;EndpointSuffix=core.windows.net;AccountName=gitlab-azure-minio-storage;AccountKey=h0tSyeTebs+..."
}
```
## Deploy Minio to Web App on Linux
First, we need to create an App Service Plan in the same resource group.
```
az appservice plan create \
--name "gitlab-azure-minio-app-plan" \
--is-linux \
--sku B1 \
--resource-group "gitlab-azure-minio" \
--location "WestUS"
```
Create a Web app configured with the [minio/minio](https://hub.docker.com/r/minio/minio) docker container, the name you specify will be used in the URL of the web app:
```
az webapp create \
--name "gitlab-minio-app" \
--deployment-container-image-name "minio/minio" \
--plan "gitlab-azure-minio-app-plan" \
--resource-group "gitlab-azure-minio"
```
The Web app should now be accessible at https://gitlab-minio-app.azurewebsites.net.
Lastly, we need to setup the startup command and create environment variables that will store our storage account name and key for use by the web app, MINIO_ACCESS_KEY & MINIO_SECRET_KEY.
```
az webapp config appsettings set \
--settings "MINIO_ACCESS_KEY=gitlab-azure-minio-storage" "MINIO_SECRET_KEY=h0tSyeTebs+..." "PORT=9000" \
--name "aleminio" \
--resource-group "Minio"
# Startup command
az webapp config set \
--startup-file "gateway azure" \
--name "gitlab-minio-app" \
--resource-group "gitlab-azure-minio"
```
## Conclusion
You can proceed to use this gateway with any client with s3-compability. Your webapp url will be the `s3 endpoint`, storage account name will be your `accesskey` and storage account key will be your `secretkey`.
## Reference
This guide was adapted for posterity from [Alessandro Segala's blog post on same topic.](https://withblue.ink/2017/10/29/how-to-use-s3cmd-and-any-other-amazon-s3-compatible-app-with-azure-blob-storage.html)
......@@ -16,6 +16,10 @@ has been provided in the [examples](https://gitlab.com/charts/gitlab/tree/master
This documentation specifies usage of access and secret keys for AWS. It is also possible to use [IAM roles](./aws-iam-roles.md).
## Azure Blob Storage
GitLab uses [fog](https://github.com/fog/fog), but [doesn't currently support fog-azure](https://gitlab.com/gitlab-org/gitlab-ce/issues/55624). To make use Azure Blob Storage, you will have to setup a [azure-minio gateway](./azure-minio-gateway.md).
## Docker Registry images
Configuration of object storage for the `registry` chart is done via the `registry.storage` key, and the `global.registry.bucket` key.
......@@ -31,13 +35,15 @@ the global is used by GitLab backups.
Create the secret per [registry chart documentation on storage](../../charts/registry/index.md#storage), then configure the chart to make use of this secret.
Examples for [S3][storage-s3](any s3 compatible) and [GCS][storage-gcs] drivers can be found in
Examples for [S3][storage-s3](any s3 compatible), [Azure][storage-azure] and [GCS][storage-gcs] drivers can be found in
[examples/objectstorage](https://gitlab.com/charts/gitlab/tree/master/examples/objectstorage).
- [registry.s3.yaml](https://gitlab.com/charts/gitlab/tree/master/examples/objectstorage/registry.s3.yaml)
- [registry.gcs.yaml](https://gitlab.com/charts/gitlab/tree/master/examples/objectstorage/registry.gcs.yaml)
- [registry.azure.yaml](https://gitlab.com/charts/gitlab/tree/master/examples/objectstorage/registry.azure.yaml)
[storage-s3]: https://docs.docker.com/registry/storage-drivers/s3
[storage-gcs]: https://docs.docker.com/registry/storage-drivers/gcs
[storage-azure]: https://docs.docker.com/registry/storage-drivers/azure
### Registry configuration
......@@ -90,13 +96,15 @@ See the [charts/globals documentaion on appConfig](../../charts/globals.md#confi
Create the secret(s) per the [connection details documentation](../../charts/globals.md#connection), and then configure the chart to use the provided secrets. Note, the same secret can be used for all of them.
Examples for [AWS][fog-aws](any S3 compatible) and [Google][fog-gcs] providers can be found in
Examples for [AWS][fog-aws](any S3 compatible like [Azure using Minio][minio-azure] ) and [Google][fog-gcs] providers can be found in
[examples/objectstorage](https://gitlab.com/charts/gitlab/tree/master/examples/objectstorage).
- [rails.s3.yaml](https://gitlab.com/charts/gitlab/tree/master/examples/objectstorage/rails.s3.yaml)
- [rails.gcs.yaml](https://gitlab.com/charts/gitlab/tree/master/examples/objectstorage/rails.gcs.yaml)
- [rails.azure.yaml](https://gitlab.com/charts/gitlab/tree/master/examples/objectstorage/rails.azure.yaml)
[fog-aws]: https://fog.io/storage/#using-amazon-s3-and-fog
[fog-gcs]: https://fog.io/storage/#google-cloud-storage
[minio-azure]: ./azure-minio-gateway.md
### appConfig configuration
......@@ -160,6 +168,27 @@ Create the secret using the [s3cmd config file format](https://s3tools.org/kb/it
enable_multipart = False
```
* On Azure Storage
```
[default]
# Setup endpoint: hostname of the Web App
host_base = https://your_minio_setup.azurewebsites.net
host_bucket = https://your_minio_setup.azurewebsites.net
# Leave as default
bucket_location = us-west-1
use_https = True
# Setup access keys
# Access Key = Azure Storage Account name
access_key = BOGUS_ACCOUNT_NAME
# Secret Key = Azure Storage Account Key
secret_key = BOGUS_KEY
# Use S3 v4 signature APIs
signature_v2 = False
```
1. Create the secret
```bash
......
# Example configuration of `connection` secret for Rails
# Example for Azure Blob Storage
# See https://gitlab.com/charts/gitlab/blob/master/doc/charts/globals.md#connection
# See https://gitlab.com/charts/gitlab/blob/master/doc/advanced/external-object-storage
# NB: GitLab uses Fog (https://github.com/fog/fog) to connect to GCS, S3 and S3 Compatible, but Azure support in Fog is on a separate project (https://github.com/fog/fog-azure-rm), which GitLab doesnt currently support, so we will use S3-Compatible for Azure.
provider: AWS
region: us-east-1
aws_access_key_id: YOUR_AZURE_STORAGE_ACCOUNT_NAME
aws_secret_access_key: YOUR_AZURE_STORAGE_ACCOUNT_KEY
# URL of your minio setup
host: your_minio_setup.azurewebsites.net
endpoint: "https://your_minio_setup.azurewebsites.net"
# Setting path_style to true is important for Azure, this will make sure your storage containers are accessed using paths and not subdomains, subdomains will fail with 404s.
path_style: true
\ No newline at end of file
# Example configuration of registry `storage` secret
# Example for Azure Blob Storage
# See https://docs.docker.com/registry/storage-drivers/azure/
# See https://gitlab.com/charts/gitlab/tree/master/doc/charts/registry/#storage
# See https://gitlab.com/charts/gitlab/blob/master/doc/advanced/external-object-storage
azure:
accountname: YOUR_AZURE_STORAGE_ACCOUNT_NAME
accountkey: YOUR_AZURE_STORAGE_ACCOUNT_KEY
container: gitlab-registry-storage
\ No newline at end of file
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment