backup_restore.md 57.1 KB
Newer Older
1
---
2
stage: Enablement
3
group: Geo
4
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
5
6
---

7
# Back up and restore GitLab **(FREE SELF)**
Marin Jankovski's avatar
Marin Jankovski committed
8

9
GitLab provides Rake tasks for backing up and restoring GitLab instances.
10

11
12
An application data backup creates an archive file that contains the database,
all repositories and all attachments.
13

14
15
You can only restore a backup to **exactly the same version and type (CE/EE)**
of GitLab on which it was created. The best way to migrate your repositories
16
from one server to another is through backup restore.
17

18
WARNING:
19
GitLab doesn't back up items that aren't stored in the file system. If you're
20
21
using [object storage](../administration/object_storage.md), be sure to enable
backups with your object storage provider, if desired.
22

23
## Requirements
24

25
26
To be able to backup and restore, ensure that Rsync is installed on your
system. If you installed GitLab:
27

28
29
- _Using the Omnibus package_, you're all set.
- _From source_, you need to determine if `rsync` is installed. For example:
30

31
32
33
  ```shell
  # Debian/Ubuntu
  sudo apt-get install rsync
34

35
36
37
  # RHEL/CentOS
  sudo yum install rsync
  ```
38

39
## Backup timestamp
40

41
42
The backup archive is saved in `backup_path`, which is specified in the
`config/gitlab.yml` file. The filename is `[TIMESTAMP]_gitlab_backup.tar`,
43
44
45
where `TIMESTAMP` identifies the time at which each backup was created, plus
the GitLab version. The timestamp is needed if you need to restore GitLab and
multiple backups are available.
46

47
For example, if the backup name is `1493107454_2018_04_25_10.6.4-ce_gitlab_backup.tar`,
48
the timestamp is `1493107454_2018_04_25_10.6.4-ce`.
49

50
## Back up GitLab
51

52
53
GitLab provides a command line interface to back up your entire instance,
including:
54
55
56
57
58
59
60
61
62

- Database
- Attachments
- Git repositories data
- CI/CD job output logs
- CI/CD job artifacts
- LFS objects
- Container Registry images
- GitLab Pages content
63
- Snippets
64
- Group wikis **(PREMIUM)**
65

66
67
68
69
70
71
Backups do not include:

- [Terraform state files](../administration/terraform_state.md)
- [Package registry files](../administration/packages/index.md)
- [Mattermost data](https://docs.mattermost.com/administration/config-settings.html#file-storage)

72
WARNING:
73
74
GitLab does not back up any configuration files, SSL certificates, or system
files. You are highly advised to read about [storing configuration files](#storing-configuration-files).
75

76
77
78
79
WARNING:
The backup command requires [additional parameters](#backup-and-restore-for-installations-using-pgbouncer) when
your installation is using PgBouncer, for either performance reasons or when using it with a Patroni cluster.

80
81
Depending on your version of GitLab, use the following command if you installed
GitLab using the Omnibus package:
82

83
84
85
86
87
- GitLab 12.2 or later:

  ```shell
  sudo gitlab-backup create
  ```
88

89
- GitLab 12.1 and earlier:
90

91
92
93
94
95
  ```shell
  gitlab-rake gitlab:backup:create
  ```

If you installed GitLab from source, use the following command:
96

97
```shell
Ben Bodenmiller's avatar
Ben Bodenmiller committed
98
sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production
99
```
100

101
102
If you're running GitLab from within a Docker container, run the backup from
the host, based on your installed version of GitLab:
103

104
105
106
107
108
- GitLab 12.2 or later:

  ```shell
  docker exec -t <container name> gitlab-backup create
  ```
109

110
- GitLab 12.1 and earlier:
111

112
  ```shell
113
  docker exec -t <container name> gitlab-rake gitlab:backup:create
114
115
116
117
118
119
  ```

If you're using the [GitLab Helm chart](https://gitlab.com/gitlab-org/charts/gitlab)
on a Kubernetes cluster, you can run the backup task by using `kubectl` to run the `backup-utility`
script on the GitLab task runner pod. For more details, see
[backing up a GitLab installation](https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/backup-restore/backup.md#backing-up-a-gitlab-installation).
120

121
```shell
122
kubectl exec -it <gitlab task-runner pod> -- backup-utility
123
124
```

125
Similar to the Kubernetes case, if you have scaled out your GitLab cluster to
126
127
use multiple application servers, you should pick a designated node (that isn't
auto-scaled away) for running the backup Rake task. Because the backup Rake
128
task is tightly coupled to the main Rails application, this is typically a node
129
on which you're also running Puma or Sidekiq.
130

131
132
Example output:

133
```plaintext
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
Dumping database tables:
- Dumping table events... [DONE]
- Dumping table issues... [DONE]
- Dumping table keys... [DONE]
- Dumping table merge_requests... [DONE]
- Dumping table milestones... [DONE]
- Dumping table namespaces... [DONE]
- Dumping table notes... [DONE]
- Dumping table projects... [DONE]
- Dumping table protected_branches... [DONE]
- Dumping table schema_migrations... [DONE]
- Dumping table services... [DONE]
- Dumping table snippets... [DONE]
- Dumping table taggings... [DONE]
- Dumping table tags... [DONE]
- Dumping table users... [DONE]
- Dumping table users_projects... [DONE]
- Dumping table web_hooks... [DONE]
- Dumping table wikis... [DONE]
Dumping repositories:
- Dumping repository abcd... [DONE]
Creating backup archive: $TIMESTAMP_gitlab_backup.tar [DONE]
Deleting tmp directories...[DONE]
Deleting old backups... [SKIPPING]
```

160
### Storing configuration files
161

162
163
164
165
166
The [backup Rake task](#back-up-gitlab) GitLab provides does _not_ store your
configuration files. The primary reason for this is that your database contains
items including encrypted information for two-factor authentication and the
CI/CD _secure variables_. Storing encrypted information in the same location
as its key defeats the purpose of using encryption in the first place.
167

168
WARNING:
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
The secrets file is essential to preserve your database encryption key.

At the very **minimum**, you must backup:

For Omnibus:

- `/etc/gitlab/gitlab-secrets.json`
- `/etc/gitlab/gitlab.rb`

For installation from source:

- `/home/git/gitlab/config/secrets.yml`
- `/home/git/gitlab/config/gitlab.yml`

For [Docker installations](https://docs.gitlab.com/omnibus/docker/), you must
184
185
186
back up the volume where the configuration files are stored. If you created
the GitLab container according to the documentation, it should be in the
`/srv/gitlab/config` directory.
187

188
189
190
191
For [GitLab Helm chart installations](https://gitlab.com/gitlab-org/charts/gitlab)
on a Kubernetes cluster, you must follow the
[Backup the secrets](https://docs.gitlab.com/charts/backup-restore/backup.html#backup-the-secrets)
instructions.
192

193
194
195
You may also want to back up any TLS keys and certificates, and your
[SSH host keys](https://superuser.com/questions/532040/copy-ssh-keys-from-one-server-to-another-server/532079#532079).

196
197
If you use Omnibus GitLab, review additional information to
[backup your configuration](https://docs.gitlab.com/omnibus/settings/backups.html).
198
199
200
201

In the unlikely event that the secrets file is lost, see the
[troubleshooting section](#when-the-secrets-file-is-lost).

202
### Backup options
203

204
205
The command line tool GitLab provides to backup your instance can accept more
options.
206

207
#### Backup strategy option
208
209
210
211
212
213

The default backup strategy is to essentially stream data from the respective
data locations to the backup using the Linux command `tar` and `gzip`. This works
fine in most cases, but can cause problems when data is rapidly changing.

When data changes while `tar` is reading it, the error `file changed as we read
214
it` may occur, and causes the backup process to fail. To combat this, 8.17
215
216
217
introduces a new backup strategy called `copy`. The strategy copies data files
to a temporary location before calling `tar` and `gzip`, avoiding the error.

218
A side-effect is that the backup process takes up to an additional 1X disk
219
220
221
222
223
space. The process does its best to clean up the temporary files at each stage
so the problem doesn't compound, but it could be a considerable change for large
installations. This is why the `copy` strategy is not the default in 8.17.

To use the `copy` strategy instead of the default streaming strategy, specify
224
225
`STRATEGY=copy` in the Rake task command. For example:

226
```shell
227
sudo gitlab-backup create STRATEGY=copy
228
```
229

230
Users of GitLab 12.1 and earlier should use the command `gitlab-rake gitlab:backup:create` instead.
231

232
233
#### Backup filename

234
WARNING:
235
If you use a custom backup filename, you can't
236
[limit the lifetime of the backups](#limit-backup-lifetime-for-local-files-prune-old-backups).
237

238
239
240
241
By default, a backup file is created according to the specification in the
previous [Backup timestamp](#backup-timestamp) section. You can, however,
override the `[TIMESTAMP]` portion of the filename by setting the `BACKUP`
environment variable. For example:
242

243
```shell
244
sudo gitlab-backup create BACKUP=dump
245
246
```

247
Users of GitLab 12.1 and earlier should use the command `gitlab-rake gitlab:backup:create` instead.
248

249
250
The resulting file is named `dump_gitlab_backup.tar`. This is useful for
systems that make use of rsync and incremental backups, and results in
251
considerably faster transfer speeds.
252

253
#### Confirm archive can be transferred
254

255
256
257
To ensure the generated archive is transferable by rsync, you can set the `GZIP_RSYNCABLE=yes`
option. This sets the `--rsyncable` option to `gzip`, which is useful only in
combination with setting [the Backup filename option](#backup-filename).
258

259
260
261
Note that the `--rsyncable` option in `gzip` isn't guaranteed to be available
on all distributions. To verify that it's available in your distribution, run
`gzip --help` or consult the man pages.
262

263
```shell
264
sudo gitlab-backup create BACKUP=dump GZIP_RSYNCABLE=yes
265
```
266

267
Users of GitLab 12.1 and earlier should use the command `gitlab-rake gitlab:backup:create` instead.
268

269
#### Excluding specific directories from the backup
270

271
You can exclude specific directories from the backup by adding the environment variable `SKIP`, whose values are a comma-separated list of the following options:
272

273
274
275
276
277
278
279
- `db` (database)
- `uploads` (attachments)
- `builds` (CI job output logs)
- `artifacts` (CI job artifacts)
- `lfs` (LFS objects)
- `registry` (Container Registry images)
- `pages` (Pages content)
280
- `repositories` (Git repositories data)
281

282
283
All wikis are backed up as part of the `repositories` group. Non-existent wikis are skipped during a backup.

284
NOTE:
285
When [backing up and restoring Helm Charts](https://docs.gitlab.com/charts/architecture/backup-restore.html), there is an additional option `packages`, which refers to any packages managed by the GitLab [package registry](../user/packages/package_registry/index.md).
286
For more information see [command line arguments](https://docs.gitlab.com/charts/architecture/backup-restore.html#command-line-arguments).
287

288
289
All wikis are backed up as part of the `repositories` group. Non-existent
wikis are skipped during a backup.
290

291
292
For Omnibus GitLab packages:

293
```shell
294
sudo gitlab-backup create SKIP=db,uploads
295
296
```

297
Users of GitLab 12.1 and earlier should use the command `gitlab-rake gitlab:backup:create` instead.
298

299
For installations from source:
300

301
```shell
302
303
304
sudo -u git -H bundle exec rake gitlab:backup:create SKIP=db,uploads RAILS_ENV=production
```

305
#### Skipping tar creation
306
307
308
309
310
311
312
313

The last part of creating a backup is generation of a `.tar` file containing
all the parts. In some cases (for example, if the backup is picked up by other
backup software) creating a `.tar` file might be wasted effort or even directly
harmful, so you can skip this step by adding `tar` to the `SKIP` environment
variable.

Adding `tar` to the `SKIP` variable leaves the files and directories containing the
314
backup in the directory used for the intermediate files. These files are
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
overwritten when a new backup is created, so you should make sure they are copied
elsewhere, because you can only have one backup on the system.

For Omnibus GitLab packages:

```shell
sudo gitlab-backup create SKIP=tar
```

For installations from source:

```shell
sudo -u git -H bundle exec rake gitlab:backup:create SKIP=tar RAILS_ENV=production
```

330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
#### Disabling prompts during restore

During a restore from backup, the restore script may ask for confirmation before
proceeding. If you wish to disable these prompts, you can set the `GITLAB_ASSUME_YES`
environment variable to `1`.

For Omnibus GitLab packages:

```shell
sudo GITLAB_ASSUME_YES=1 gitlab-backup restore
```

For installations from source:

```shell
sudo -u git -H GITLAB_ASSUME_YES=1 bundle exec rake gitlab:backup:restore RAILS_ENV=production
```

348
349
#### Back up Git repositories concurrently

350
351
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37158) in GitLab 13.3.
> - [Concurrent restore introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/69330) in GitLab 14.3
352

353
When using [multiple repository storages](../administration/repository_storage_paths.md),
354
repositories can be backed up or restored concurrently to help fully use CPU time. The
355
356
following variables are available to modify the default behavior of the Rake
task:
357

358
- `GITLAB_BACKUP_MAX_CONCURRENCY`: The maximum number of projects to back up at
359
360
  the same time. Defaults to the number of logical CPUs (in GitLab 14.1 and
  earlier, defaults to `1`).
361
362
- `GITLAB_BACKUP_MAX_STORAGE_CONCURRENCY`: The maximum number of projects to
  back up at the same time on each storage. This allows the repository backups
363
364
  to be spread across storages. Defaults to `2` (in GitLab 14.1 and earlier,
  defaults to `1`).
365

366
For example, for Omnibus GitLab installations with 4 repository storages:
367
368
369
370
371
372
373
374
375
376
377

```shell
sudo gitlab-backup create GITLAB_BACKUP_MAX_CONCURRENCY=4 GITLAB_BACKUP_MAX_STORAGE_CONCURRENCY=1
```

For example, for installations from source:

```shell
sudo -u git -H bundle exec rake gitlab:backup:create GITLAB_BACKUP_MAX_CONCURRENCY=4 GITLAB_BACKUP_MAX_STORAGE_CONCURRENCY=1
```

378
#### Uploading backups to a remote (cloud) storage
379

380
381
382
You can let the backup script upload (using the [Fog library](http://fog.io/))
the `.tar` file it creates. In the following example, we use Amazon S3 for
storage, but Fog also lets you use [other storage providers](http://fog.io/storage/).
383
GitLab also [imports cloud drivers](https://gitlab.com/gitlab-org/gitlab/-/blob/da46c9655962df7d49caef0e2b9f6bbe88462a02/Gemfile#L113)
384
for AWS, Google, OpenStack Swift, Rackspace, and Aliyun. A local driver is
385
[also available](#uploading-to-locally-mounted-shares).
386

387
388
[Read more about using object storage with GitLab](../administration/object_storage.md).

389
##### Using Amazon S3
390

391
392
393
394
For Omnibus GitLab packages:

1. Add the following to `/etc/gitlab/gitlab.rb`:

395
396
397
398
399
400
401
402
403
404
405
   ```ruby
   gitlab_rails['backup_upload_connection'] = {
     'provider' => 'AWS',
     'region' => 'eu-west-1',
     'aws_access_key_id' => 'AKIAKIAKI',
     'aws_secret_access_key' => 'secret123'
     # If using an IAM Profile, don't configure aws_access_key_id & aws_secret_access_key
     # 'use_iam_profile' => true
   }
   gitlab_rails['backup_upload_remote_directory'] = 'my.s3.bucket'
   ```
406

407
408
1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure)
   for the changes to take effect
409

410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
##### S3 Encrypted Buckets

AWS supports these [modes for server side encryption](https://docs.aws.amazon.com/AmazonS3/latest/userguide/serv-side-encryption.html):

- Amazon S3-Managed Keys (SSE-S3)
- Customer Master Keys (CMKs) Stored in AWS Key Management Service (SSE-KMS)
- Customer-Provided Keys (SSE-C)

Use your mode of choice with GitLab. Each mode has similar, but slightly
different, configuration methods.

###### SSE-S3

To enable SSE-S3, in the backup storage options set the `server_side_encryption`
field to `AES256`. For example, in Omnibus GitLab:

```ruby
gitlab_rails['backup_upload_storage_options'] = {
  'server_side_encryption' => 'AES256'
}
```

###### SSE-KMS

To enable SSE-KMS, you'll need the [KMS key via its Amazon Resource Name (ARN)
in the `arn:aws:kms:region:acct-id:key/key-id` format](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingKMSEncryption.html). Under the `backup_upload_storage_options` config setting, set:

- `server_side_encryption` to `aws:kms`.
- `server_side_encryption_kms_key_id` to the ARN of the key.

For example, in Omnibus GitLab:

```ruby
gitlab_rails['backup_upload_storage_options'] = {
  'server_side_encryption' => 'aws:kms',
  'server_side_encryption_kms_key_id' => 'arn:aws:<YOUR KMS KEY ID>:'
}
```

###### SSE-C

SSE-C requires you to set these encryption options:

- `backup_encryption`: AES256.
- `backup_encryption_key`: Unencoded, 32-byte (256 bits) key. The upload fails if this isn't exactly 32 bytes.

For example, in Omnibus GitLab:

```ruby
gitlab_rails['backup_encryption'] = 'AES256'
gitlab_rails['backup_encryption_key'] = '<YOUR 32-BYTE KEY HERE>'
```

If the key contains binary characters and cannot be encoded in UTF-8,
instead, specify the key with the `GITLAB_BACKUP_ENCRYPTION_KEY` environment variable.
For example:

```ruby
gitlab_rails['env'] = { 'GITLAB_BACKUP_ENCRYPTION_KEY' => "\xDE\xAD\xBE\xEF" * 8 }
```

471
##### Digital Ocean Spaces
472

473
This example can be used for a bucket in Amsterdam (AMS3):
474

475
1. Add the following to `/etc/gitlab/gitlab.rb`:
476

477
478
479
480
481
482
483
484
485
486
   ```ruby
   gitlab_rails['backup_upload_connection'] = {
     'provider' => 'AWS',
     'region' => 'ams3',
     'aws_access_key_id' => 'AKIAKIAKI',
     'aws_secret_access_key' => 'secret123',
     'endpoint'              => 'https://ams3.digitaloceanspaces.com'
   }
   gitlab_rails['backup_upload_remote_directory'] = 'my.s3.bucket'
   ```
487

488
489
1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure)
   for the changes to take effect
490

491
492
493
494
If you see a `400 Bad Request` error message when using Digital Ocean Spaces,
the cause may be the use of backup encryption. Because Digital Ocean Spaces
doesn't support encryption, remove or comment the line that contains
`gitlab_rails['backup_encryption']`.
495

496
##### Other S3 Providers
497

498
499
500
501
Not all S3 providers are fully compatible with the Fog library. For example,
if you see a `411 Length Required` error message after attempting to upload,
you may need to downgrade the `aws_signature_version` value from the default
value to `2`, [due to this issue](https://github.com/fog/fog-aws/issues/428).
502

503
504
For installations from source:

505
506
1. Edit `home/git/gitlab/config/gitlab.yml`:

507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
   ```yaml
     backup:
       # snip
       upload:
         # Fog storage connection settings, see http://fog.io/storage/ .
         connection:
           provider: AWS
           region: eu-west-1
           aws_access_key_id: AKIAKIAKI
           aws_secret_access_key: 'secret123'
           # If using an IAM Profile, leave aws_access_key_id & aws_secret_access_key empty
           # ie. aws_access_key_id: ''
           # use_iam_profile: 'true'
         # The remote 'directory' to store your backups. For S3, this would be the bucket name.
         remote_directory: 'my.s3.bucket'
         # Specifies Amazon S3 storage class to use for backups, this is optional
         # storage_class: 'STANDARD'
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
         #
         # Turns on AWS Server-Side Encryption with Amazon Customer-Provided Encryption Keys for backups, this is optional
         #   'encryption' must be set in order for this to have any effect.
         #   'encryption_key' should be set to the 256-bit encryption key for Amazon S3 to use to encrypt or decrypt.
         #   To avoid storing the key on disk, the key can also be specified via the `GITLAB_BACKUP_ENCRYPTION_KEY`  your data.
         # encryption: 'AES256'
         # encryption_key: '<key>'
         #
         #
         # Turns on AWS Server-Side Encryption with Amazon S3-Managed keys (optional)
         # https://docs.aws.amazon.com/AmazonS3/latest/userguide/serv-side-encryption.html
         # For SSE-S3, set 'server_side_encryption' to 'AES256'.
         # For SS3-KMS, set 'server_side_encryption' to 'aws:kms'. Set
         # 'server_side_encryption_kms_key_id' to the ARN of customer master key.
         # storage_options:
         #   server_side_encryption: 'aws:kms'
         #   server_side_encryption_kms_key_id: 'arn:aws:kms:YOUR-KEY-ID-HERE'
541
   ```
542

543
544
1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source)
   for the changes to take effect
545

546
If you're uploading your backups to S3, you should create a new
547
IAM user with restricted access rights. To give the upload user access only for
548
uploading backups create the following IAM profile, replacing `my.s3.bucket`
549
550
551
552
with the name of your bucket:

```json
{
553
  "Version": "2012-10-17",
554
555
  "Statement": [
    {
556
      "Sid": "Stmt1412062044000",
557
558
559
560
561
562
563
      "Effect": "Allow",
      "Action": [
        "s3:AbortMultipartUpload",
        "s3:GetBucketAcl",
        "s3:GetBucketLocation",
        "s3:GetObject",
        "s3:GetObjectAcl",
564
        "s3:ListBucketMultipartUploads",
565
566
567
568
569
570
        "s3:PutObject",
        "s3:PutObjectAcl"
      ],
      "Resource": [
        "arn:aws:s3:::my.s3.bucket/*"
      ]
571
    },
572
    {
573
      "Sid": "Stmt1412062097000",
574
575
      "Effect": "Allow",
      "Action": [
576
577
        "s3:GetBucketLocation",
        "s3:ListAllMyBuckets"
578
579
580
581
      ],
      "Resource": [
        "*"
      ]
582
    },
583
    {
584
      "Sid": "Stmt1412062128000",
585
586
587
588
589
590
591
592
593
594
595
596
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::my.s3.bucket"
      ]
    }
  ]
}
```

597
##### Using Google Cloud Storage
598

599
600
To use Google Cloud Storage to save backups, you must first create an
access key from the Google console:
601

602
603
604
605
606
607
608
1. Go to the [Google storage settings page](https://console.cloud.google.com/storage/settings).
1. Select **Interoperability**, and then create an access key.
1. Make note of the **Access Key** and **Secret** and replace them in the
   following configurations.
1. In the buckets advanced settings ensure the Access Control option
   **Set object-level and bucket-level permissions** is selected.
1. Ensure you have already created a bucket.
609
610
611
612
613

For Omnibus GitLab packages:

1. Edit `/etc/gitlab/gitlab.rb`:

614
615
616
617
   ```ruby
   gitlab_rails['backup_upload_connection'] = {
     'provider' => 'Google',
     'google_storage_access_key_id' => 'Access Key',
618
619
620
621
622
623
624
     'google_storage_secret_access_key' => 'Secret',

     ## If you have CNAME buckets (foo.example.com), you might run into SSL issues
     ## when uploading backups ("hostname foo.example.com.storage.googleapis.com
     ## does not match the server certificate"). In that case, uncomnent the following
     ## setting. See: https://github.com/fog/fog/issues/2834
     #'path_style' => true
625
626
627
   }
   gitlab_rails['backup_upload_remote_directory'] = 'my.google.bucket'
   ```
628

629
630
1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure)
   for the changes to take effect
631
632
633
634
635

For installations from source:

1. Edit `home/git/gitlab/config/gitlab.yml`:

636
637
638
639
640
641
642
643
644
   ```yaml
     backup:
       upload:
         connection:
           provider: 'Google'
           google_storage_access_key_id: 'Access Key'
           google_storage_secret_access_key: 'Secret'
         remote_directory: 'my.google.bucket'
   ```
645

646
647
1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source)
   for the changes to take effect
648

649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
##### Using Azure Blob storage

> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/25877) in GitLab 13.4.

For Omnibus GitLab packages:

1. Edit `/etc/gitlab/gitlab.rb`:

   ```ruby
   gitlab_rails['backup_upload_connection'] = {
    'provider' => 'AzureRM',
    'azure_storage_account_name' => '<AZURE STORAGE ACCOUNT NAME>',
    'azure_storage_access_key' => '<AZURE STORAGE ACCESS KEY>',
    'azure_storage_domain' => 'blob.core.windows.net', # Optional
   }
   gitlab_rails['backup_upload_remote_directory'] = '<AZURE BLOB CONTAINER>'
   ```

667
668
1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure)
   for the changes to take effect
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683

For installations from source:

1. Edit `home/git/gitlab/config/gitlab.yml`:

   ```yaml
     backup:
       upload:
         connection:
           provider: 'AzureRM'
           azure_storage_account_name: '<AZURE STORAGE ACCOUNT NAME>'
           azure_storage_access_key: '<AZURE STORAGE ACCESS KEY>'
         remote_directory: '<AZURE BLOB CONTAINER>'
   ```

684
685
1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source)
   for the changes to take effect
686

687
For more details, see the [table of Azure parameters](../administration/object_storage.md#azure-blob-storage).
688

689
##### Specifying a custom directory for backups
690

691
This option works only for remote storage. If you want to group your backups,
692
693
you can pass a `DIRECTORY` environment variable:

694
```shell
695
696
sudo gitlab-backup create DIRECTORY=daily
sudo gitlab-backup create DIRECTORY=weekly
697
698
```

699
Users of GitLab 12.1 and earlier should use the command `gitlab-rake gitlab:backup:create` instead.
700

701
#### Uploading to locally mounted shares
702

703
704
705
706
707
708
You may also send backups to a mounted share (for example, `NFS`,`CIFS`, or
`SMB`) by using the Fog [`Local`](https://github.com/fog/fog-local#usage)
storage provider. The directory pointed to by the `local_root` key _must_ be
owned by the `git` user _when mounted_ (mounting with the `uid=` of the `git`
user for `CIFS` and `SMB`) or the user that you are executing the backup tasks
as (for Omnibus packages, this is the `git` user).
709

710
The `backup_upload_remote_directory` _must_ be set in addition to the
711
`local_root` key. This is the sub directory inside the mounted directory that
712
backups are copied to, and is created if it does not exist. If the
713
directory that you want to copy the tarballs to is the root of your mounted
714
directory, use `.` instead.
715

716
Because file system performance may affect overall GitLab performance,
717
[GitLab doesn't recommend using cloud-based file systems for storage](../administration/nfs.md#avoid-using-cloud-based-file-systems).
718

719
For Omnibus GitLab packages:
720

721
722
1. Edit `/etc/gitlab/gitlab.rb`:

723
724
725
726
727
   ```ruby
   gitlab_rails['backup_upload_connection'] = {
     :provider => 'Local',
     :local_root => '/mnt/backups'
   }
728

729
730
731
732
   # The directory inside the mounted folder to copy backups to
   # Use '.' to store them in the root directory
   gitlab_rails['backup_upload_remote_directory'] = 'gitlab_backups'
   ```
733

734
735
1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure)
   for the changes to take effect.
736

737
738
For installations from source:

739
740
1. Edit `home/git/gitlab/config/gitlab.yml`:

741
742
743
744
745
746
747
748
749
750
751
   ```yaml
   backup:
     upload:
       # Fog storage connection settings, see http://fog.io/storage/ .
       connection:
         provider: Local
         local_root: '/mnt/backups'
       # The directory inside the mounted folder to copy backups to
       # Use '.' to store them in the root directory
       remote_directory: 'gitlab_backups'
   ```
752

753
754
1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source)
   for the changes to take effect.
755

756
#### Backup archive permissions
757

758
The backup archives created by GitLab (`1393513186_2014_02_27_gitlab_backup.tar`)
759
have the owner/group `git`/`git` and 0600 permissions by default. This is
760
meant to avoid other system users reading GitLab data. If you need the backup
761
762
archives to have different permissions, you can use the `archive_permissions`
setting.
763

764
For Omnibus GitLab packages:
765

766
1. Edit `/etc/gitlab/gitlab.rb`:
767

768
769
770
   ```ruby
   gitlab_rails['backup_archive_permissions'] = 0644 # Makes the backup archives world-readable
   ```
771

772
773
1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure)
   for the changes to take effect.
774
775

For installations from source:
776

777
1. Edit `/home/git/gitlab/config/gitlab.yml`:
778

779
780
781
782
   ```yaml
   backup:
     archive_permissions: 0644 # Makes the backup archives world-readable
   ```
783

784
785
1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source)
   for the changes to take effect.
786

787
#### Configuring cron to make daily backups
788

789
WARNING:
790
791
792
The following cron jobs do not [backup your GitLab configuration files](#storing-configuration-files)
or [SSH host keys](https://superuser.com/questions/532040/copy-ssh-keys-from-one-server-to-another-server/532079#532079).

793
794
You can schedule a cron job that backs up your repositories and GitLab metadata.

795
796
For Omnibus GitLab packages:

797
1. Edit the crontab for the `root` user:
798

799
800
801
   ```shell
   sudo su -
   crontab -e
802
   ```
803

804
1. There, add the following line to schedule the backup for everyday at 2 AM:
805

806
807
808
   ```plaintext
   0 2 * * * /opt/gitlab/bin/gitlab-backup create CRON=1
   ```
809

810
   Users of GitLab 12.1 and earlier should use the command `gitlab-rake gitlab:backup:create` instead.
811

812
For installations from source:
813

814
1. Edit the crontab for the `git` user:
815

816
817
818
   ```shell
   sudo -u git crontab -e
   ```
819

820
821
822
823
824
825
826
1. Add the following lines at the bottom:

   ```plaintext
   # Create a full backup of the GitLab repositories and SQL database every day at 2am
   0 2 * * * cd /home/git/gitlab && PATH=/usr/local/bin:/usr/bin:/bin bundle exec rake gitlab:backup:create RAILS_ENV=production CRON=1
   ```

827
828
The `CRON=1` environment setting directs the backup script to hide all progress
output if there aren't any errors. This is recommended to reduce cron spam.
829
830
831

### Limit backup lifetime for local files (prune old backups)

832
WARNING:
833
The process described in this section don't work if you used a [custom filename](#backup-filename)
834
835
for your backups.

836
837
838
839
840
841
842
To prevent regular backups from using all your disk space, you may want to set a limited lifetime
for backups. The next time the backup task runs, backups older than the `backup_keep_time` are
pruned.

This configuration option manages only local files. GitLab doesn't prune old
files stored in a third-party [object storage](#uploading-backups-to-a-remote-cloud-storage)
because the user may not have permission to list and delete files. It's
843
844
recommended that you configure the appropriate retention policy for your object
storage (for example, [AWS S3](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-lifecycle.html)).
845

846
847
848
849
850
851
852
853
854
For Omnibus GitLab packages:

1. Edit `/etc/gitlab/gitlab.rb`:

   ```ruby
   ## Limit backup lifetime to 7 days - 604800 seconds
   gitlab_rails['backup_keep_time'] = 604800
   ```

855
856
1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure)
   for the changes to take effect.
857

858
859
For installations from source:

860
1. Edit `/home/git/gitlab/config/gitlab.yml`:
861

862
863
864
865
866
   ```yaml
   backup:
     ## Limit backup lifetime to 7 days - 604800 seconds
     keep_time: 604800
   ```
867

868
869
1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source)
   for the changes to take effect.
870

871
## Restore GitLab
872

873
GitLab provides a command line interface to restore your entire installation,
874
875
876
and is flexible enough to fit your needs.

The [restore prerequisites section](#restore-prerequisites) includes crucial
877
878
information. Be sure to read and test the complete restore process at least
once before attempting to perform it in a production environment.
879

880
881
You can restore a backup only to _the exact same version and type (CE/EE)_ of
GitLab that you created it on (for example CE 9.1.0).
882

883
If your backup is a different version than the current installation, you must
884
[downgrade your GitLab installation](../update/package/downgrade.md)
885
886
before restoring the backup.

887
### Restore prerequisites
888

889
890
891
You need to have a working GitLab installation before you can perform a
restore. This is because the system user performing the restore actions (`git`)
is usually not allowed to create or delete the SQL database needed to import
892
data into (`gitlabhq_production`). All existing data is either erased
893
(SQL) or moved to a separate directory (such as repositories and uploads).
894

895
To restore a backup, you must restore `/etc/gitlab/gitlab-secrets.json`
896
897
(for Omnibus packages) or `/home/git/gitlab/.secret` (for installations from
source). This file contains the database encryption key,
898
[CI/CD variables](../ci/variables/index.md), and
899
variables used for [two-factor authentication](../user/profile/account/two_factor_authentication.md).
900
If you fail to restore this encryption key file along with the application data
901
902
backup, users with two-factor authentication enabled and GitLab Runner
loses access to your GitLab server.
903

904
905
You may also want to restore any TLS keys, certificates, or
[SSH host keys](https://superuser.com/questions/532040/copy-ssh-keys-from-one-server-to-another-server/532079#532079).
Davin Walker's avatar
Davin Walker committed
906

907
Starting with GitLab 12.9, if an untarred backup (like the ones made with
908
909
910
`SKIP=tar`) is found, and no backup is chosen with `BACKUP=<timestamp>`, the
untarred backup is used.

911
912
913
Depending on your case, you might want to run the restore command with one or
more of the following options:

914
- `BACKUP=timestamp_of_backup`: Required if more than one backup exists.
915
  Read what the [backup timestamp is about](#backup-timestamp).
916
- `force=yes`: Doesn't ask if the authorized_keys file should get regenerated,
917
  and assumes 'yes' for warning about database tables being removed,
918
919
  enabling the "Write to authorized_keys file" setting, and updating LDAP
  providers.
920

921
922
923
If you're restoring into directories that are mount points, you must ensure these directories are
empty before attempting a restore. Otherwise, GitLab attempts to move these directories before
restoring the new data, which causes an error.
924

925
Read more about [configuring NFS mounts](../administration/nfs.md)
926

927
### Restore for Omnibus GitLab installations
Valeriy's avatar
Valeriy committed
928

929
This procedure assumes that:
Valeriy's avatar
Valeriy committed
930

931
- You have installed the **exact same version and type (CE/EE)** of GitLab
932
  Omnibus with which the backup was created.
933
- You have run `sudo gitlab-ctl reconfigure` at least once.
934
- GitLab is running. If not, start it using `sudo gitlab-ctl start`.
935

936
First ensure your backup tar file is in the backup directory described in the
937
`gitlab.rb` configuration `gitlab_rails['backup_path']`. The default is
938
`/var/opt/gitlab/backups`. It needs to be owned by the `git` user.
Valeriy's avatar
Valeriy committed
939
940

```shell
941
sudo cp 11493107454_2018_04_25_10.6.4-ce_gitlab_backup.tar /var/opt/gitlab/backups/
Craig Fisher's avatar
Craig Fisher committed
942
sudo chown git.git /var/opt/gitlab/backups/11493107454_2018_04_25_10.6.4-ce_gitlab_backup.tar
Valeriy's avatar
Valeriy committed
943
944
```

945
Stop the processes that are connected to the database. Leave the rest of GitLab
946
running:
Valeriy's avatar
Valeriy committed
947
948

```shell
949
sudo gitlab-ctl stop puma
Valeriy's avatar
Valeriy committed
950
sudo gitlab-ctl stop sidekiq
951
952
953
# Verify
sudo gitlab-ctl status
```
Valeriy's avatar
Valeriy committed
954

955
956
957
958
Next, restore the backup, specifying the timestamp of the backup you wish to
restore:

```shell
Valeriy's avatar
Valeriy committed
959
# This command will overwrite the contents of your GitLab database!
960
sudo gitlab-backup restore BACKUP=11493107454_2018_04_25_10.6.4-ce
961
```
Valeriy's avatar
Valeriy committed
962

963
Users of GitLab 12.1 and earlier should use the command `gitlab-rake gitlab:backup:restore` instead.
964

965
WARNING:
966
967
`gitlab-rake gitlab:backup:restore` doesn't set the correct file system
permissions on your Registry directory. This is a [known issue](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/62759).
968
In GitLab 12.2 or later, you can use `gitlab-backup restore` to avoid this
969
issue.
970

971
972
973
974
975
If there's a GitLab version mismatch between your backup tar file and the
installed version of GitLab, the restore command aborts with an error
message. Install the [correct GitLab version](https://packages.gitlab.com/gitlab/),
and then try again.

976
977
978
WARNING:
The restore command requires [additional parameters](#backup-and-restore-for-installations-using-pgbouncer) when
your installation is using PgBouncer, for either performance reasons or when using it with a Patroni cluster.
979
980
981

Next, restore `/etc/gitlab/gitlab-secrets.json` if necessary,
[as previously mentioned](#restore-prerequisites).
982

983
Reconfigure, restart and check GitLab:
Valeriy's avatar
Valeriy committed
984

985
```shell
986
sudo gitlab-ctl reconfigure
987
sudo gitlab-ctl restart
Valeriy's avatar
Valeriy committed
988
989
990
sudo gitlab-rake gitlab:check SANITIZE=true
```

991
In GitLab 13.1 and later, check [database values can be decrypted](../administration/raketasks/doctor.md)
992
993
especially if `/etc/gitlab/gitlab-secrets.json` was restored, or if a different server is
the target for the restore.
Valeriy's avatar
Valeriy committed
994

995
996
997
```shell
sudo gitlab-rake gitlab:doctor:secrets
```
998

999
### Restore for Docker image and GitLab Helm chart installations
1000

1001
1002
1003
1004
1005
1006
1007
1008
For GitLab installations using the Docker image or the GitLab Helm chart on a
Kubernetes cluster, the restore task expects the restore directories to be
empty. However, with Docker and Kubernetes volume mounts, some system level
directories may be created at the volume roots, such as the `lost+found`
directory found in Linux operating systems. These directories are usually owned
by `root`, which can cause access permission errors since the restore Rake task
runs as the `git` user. To restore a GitLab installation, users have to confirm
the restore target directories are empty.
1009

1010
1011
For both these installation types, the backup tarball has to be available in
the backup location (default location is `/var/opt/gitlab/backups`).
1012

1013
For Docker installations, the restore task can be run from host:
1014

1015
```shell
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
# Stop the processes that are connected to the database
docker exec -it <name of container> gitlab-ctl stop puma
docker exec -it <name of container> gitlab-ctl stop sidekiq

# Verify that the processes are all down before continuing
docker exec -it <name of container> gitlab-ctl status

# Run the restore
docker exec -it <name of container> gitlab-backup restore BACKUP=11493107454_2018_04_25_10.6.4-ce

# Restart the GitLab container
docker restart <name of container>

# Check GitLab
docker exec -it <name of container> gitlab-rake gitlab:check SANITIZE=true
1031
1032
```

1033
Users of GitLab 12.1 and earlier should use the command `gitlab-rake gitlab:backup:create` instead.
1034

1035
WARNING:
1036
1037
`gitlab-rake gitlab:backup:restore` doesn't set the correct file system
permissions on your Registry directory. This is a [known issue](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/62759).
1038
In GitLab 12.2 or later, you can use `gitlab-backup restore` to avoid this
1039
issue.
1040

1041
1042
The GitLab Helm chart uses a different process, documented in