Commit a5730050 authored by Vic Iglesias's avatar Vic Iglesias

Add docs for GCS backups

parent f54cfc09
Pipeline #53940724 passed with stages
in 1 minute and 39 seconds
......@@ -73,7 +73,7 @@ via the `global.appConfig.lfs`, `global.appConfig.artifacts`, `global.appConfig.
--set global.appConfig.pseudonymizer.bucket=gitlab-pseudonymizer-storage
--set global.appConfig.pseudonymizer.connection.secret=object-storage
--set global.appConfig.pseudonymizer.connection.key=connection
````
```
> **Note**: Currently a different bucket is needed for each, otherwise performing a restore from backup will not properly function.
......@@ -108,22 +108,20 @@ A connection configuration through the `gitlab.task-runner.backups.objectStorage
```
--set global.appConfig.backups.bucket=gitlab-backup-storage
--set global.appConfig.backups.tmpBucket=gitlab-tmp-storage
--set gitlab.task-runner.backups.objectStorage.config.secret=s3cmd-config
--set gitlab.task-runner.backups.objectStorage.config.secret=storage-config
--set gitlab.task-runner.backups.objectStorage.config.key=config
```
See the [backup/restore object storage documentation](../../backup-restore/index.md#object-storage) for full details.
Create the secret using the [s3cmd config file format](https://s3tools.org/kb/item14.htm) per the documentation.
> **Note**: In order to backup/restore files from the other object storage locations, the s3cmd config file needs to be
> **Note**: In order to backup/restore files from the other object storage locations, the config file needs to be
> configured to authenticate as a user with sufficient access to read/write to all GitLab buckets.
### Backups storage example
1. Create a file called `s3cmd.config` containing:
1. Create the `storage.config` file:
* On Amazon S3
* On Amazon S3, the contents should be in the [s3cmd config file format](https://s3tools.org/kb/item14.htm)
```
[default]
......@@ -132,27 +130,20 @@ Create the secret using the [s3cmd config file format](https://s3tools.org/kb/it
bucket_location = us-east-1
```
* On Google Cloud Storage
```
[default]
host_base = storage.googleapis.com
host_bucket = storage.googleapis.com
use_https = True
signature_v2 = True
# Access and secret key can be generated in the interoperability
# https://console.cloud.google.com/storage/settings
# See Docs: https://cloud.google.com/storage/docs/interoperability
access_key = BOGUS_ACCESS_KEY
secret_key = BOGUS_SECRET_KEY
* On Google Cloud Storage, you can create the file by creating a service account
with the storage.admin role and then
[creating a service account key](https://cloud.google.com/iam/docs/creating-managing-service-account-keys#creating_service_account_keys).
Below is an example of using the `gcloud` CLI to create the file.
# Multipart needs to be disabled for GCS !
enable_multipart = False
```shell
export PROJECT_ID=$(gcloud config get-value project)
gcloud iam service-accounts create gitlab-gcs --display-name "Gitlab Cloud Storage"
gcloud projects add-iam-policy-binding --role roles/storage.admin ${PROJECT_ID} --member=serviceAccount:gitlab-gcs@${PROJECT_ID}.iam.gserviceaccount.com
gcloud iam service-accounts keys create --iam-account gitlab-gcs@${PROJECT_ID}.iam.gserviceaccount.com storage.config
```
1. Create the secret
```bash
kubectl create secret generic s3cmd-config --from-file=config=s3cmd.config
kubectl create secret generic storage-config --from-file=config=storage.config
```
......@@ -40,6 +40,13 @@ The sequence of execution is:
The default name of the bucket that will be used to store backups is `gitlab-backups`. This is configurable
using the `BACKUP_BUCKET_NAME` environment variable.
#### Backing up to Google Cloud Storage
By default, the backup utility uses `s3cmd` to upload and download artifacts from object storage. While this can work with Google Cloud Storage (GCS),
it requires using the Interoperability API which makes undesireable compromises to authentication and authorization. When using Google Cloud Storage
for backups you can configure the backup utility script to use the Cloud Storage native CLI, `gsutil`, to do the upload and download
of your artifacts by setting the `BACKUP_BACKEND` environment variable to `gcs`.
### Restore
The backup utility when given an argument `--restore` attempts to restore from an existing backup to the running instance. This
......
......@@ -12,7 +12,9 @@ Technical details for how the utility works can be found in the [architecture do
## Object storage
We provide a minio instance out of the box when using this charts unless an [external object storage](../advanced/external-object-storage/index.md) is specified. The default behavior of the task-runner pod defaults to connect to our minio unless specific settings are given.
We provide a minio instance out of the box when using this charts unless an [external object storage](../advanced/external-object-storage/index.md) is specified. The default behavior of the task-runner pod defaults to connect to our minio unless specific settings are given. The task-runner can also be configured to back up to Amazon S3 or Google Cloud Storage (GCS).
### Backups to S3
The task-runner uses `s3cmd` to connect to object storage. In order to configure connectivity to external object storage `gitlab.task-runner.backups.objectStorage.config.secret` should be specified which points to a kubernetes secret containing a `.s3cfg` file. `gitlab.task-runner.backups.objectStorage.config.key` should be specified if different from the default of `config`. This points to the key containing the contents of a .s3cfg file.
......@@ -34,6 +36,36 @@ when restoring a backup.
--set global.appConfig.backups.tmpBucket=gitlab-tmp-storage
```
### Backups to Google Cloud Storage (GCS)
To backup to GCS you must set `gitlab.task-runner.backups.objectStorage.backend` to `gcs`. This ensures that the task-runner uses the `gsutil` CLI when storing and retrieving
objects. You must create a Kubernetes secret with the contents of an active service account JSON key where the service account has the `storage.admin` role for the buckets
you will use for backup. Below is an example of using the `gcloud` and `kubectl` to create the secret.
```shell
export PROJECT_ID=$(gcloud config get-value project)
gcloud iam service-accounts create gitlab-gcs --display-name "Gitlab Cloud Storage"
gcloud projects add-iam-policy-binding --role roles/storage.admin ${PROJECT_ID} --member=serviceAccount:gitlab-gcs@${PROJECT_ID}.iam.gserviceaccount.com
gcloud iam service-accounts keys create --iam-account gitlab-gcs@${PROJECT_ID}.iam.gserviceaccount.com storage.config
kubectl create secret generic storage-config --from-file=config=storage.config
```
Configure your Helm chart as follows to use the service account key to authenticate for backups:
```sh
helm install gitlab \
--set gitlab.task-runner.backups.objectStorage.config.secret=storage-config \
--set gitlab.task-runner.backups.objectStorage.config.key=config .
```
In addition, two bucket locations need to be configured, one for storing the backups, and one temporary bucket that is used
when restoring a backup.
```
--set global.appConfig.backups.bucket=gitlab-backup-storage
--set global.appConfig.backups.tmpBucket=gitlab-tmp-storage
```
## Backup and Restoring procedures
- [Backing up a GitLab installation](backup.md)
......
......@@ -218,6 +218,19 @@ See [nginx-ingress chart](../charts/nginx/index.md)
| gitlab.sidekiq.timeout | Sidekiq job timeout | 5 |
| gitlab.sidekiq.resources.requests.cpu | Sidekiq minimum needed cpu | 100m |
| gitlab.sidekiq.resources.requests.memory | Sidekiq minimum needed memory | 600M |
| gitlab.task-runner.enabled | Task runner enabled flag | true |
| gitlab.task-runner.image.repository | Task runner image repository | registry.gitlab.com/gitlab-org/build/cng/gitlab-task-runner-ee |
| gitlab.task-runner.image.tag | Task runner image tag | latest |
| gitlab.task-runner.image.pullPolicy | Task runner image pull policy | IfNotPresent |
| gitlab.task-runner.init.image | Task runner init image repository | busybox |
| gitlab.task-runner.init.tag | Task runner init image tag | latest |
| gitlab.task-runner.init.resources.requests.cpu | Task runner init minimum needed cpu | 50m |
| gitlab.task-runner.annotations | Annotations to add to the task runner | {} |
| gitlab.task-runner.backups.objectStorage.backend | Object storage provider to use (`s3` or `gcs`) | s3 |
| gitlab.task-runner.backups.objectStorage.config | Authentication information for object storage | {} |
| gitlab.task-runner.backups.objectStorage.config.secret | Object storage credentials secret | "" |
| gitlab.task-runner.backups.objectStorage.config.key | key containing credentials in secret | "" |
| gitlab.task-runner.backups.objectStorage.config.gcpProject | GCP Project to use when backend is `gcs` | "" |
| gitlab.unicorn.replicaCount | Unicorn number of replicas | 1 |
| gitlab.unicorn.image.repository | Unicorn image repository | registry.gitlab.com/gitlab-org/build/cng/gitlab-unicorn-ee |
| gitlab.unicorn.image.tag | Unicorn image tag | latest |
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment