Skip to content

Add AWS SSE-KMS support for backups

Stan Hu requested to merge sh-s3-aws-kms-backup-encryption into master

This adds support for the AWS SSE-KMS configuration parameters for backups added in gitlab!64765 (merged). For example:

gitlab_rails['backup_upload_storage_options'] = {
  'server_side_encryption' => 'aws:kms',
  'server_side_encryption_kms_key_id' => 'arn:aws:12345'
}

As described in https://docs.aws.amazon.com/AmazonS3/latest/userguide/serv-side-encryption.html, AWS supports three different modes for encrypting S3 data:

  1. Server-Side Encryption with Amazon S3-Managed Keys (SSE-S3)
  2. Server-Side Encryption with Customer Master Keys (CMKs) Stored in AWS Key Management Service (SSE-KMS)
  3. Server-Side Encryption with Customer-Provided Keys (SSE-C)

Previously, SSE-S3 and SSE-C were supported via the backup.upload.encryption and backup.upload.encryption_key configuration options.

SSE-KMS was previously not supported in backups because there was no way to specify which customer-managed key to use. However, we did support SSE-KMS with consolidated object storage enabled for other CI artifacts, attachments, LFS, etc. Note that SSE-C is NOT supported here.

In consolidated object storage, the storage_options Hash provides the server_side_encryption and server_side_encryption_kms_key_id parameters that allow admins to configure SSE-KMS. We reuse this configuration in backups to support SSE-KMS.

Related issues

Relates to gitlab#338764 (closed)

Testing

Before

With an S3 bucket policy to reject unencrypted uploads (gitlab!64765 (merged)):

$ sudo gitlab-rake gitlab:backup:create SKIP=uploads,builds,artifacts,lfs,registry,pages,repositories
2021-09-07 06:35:24 +0000 -- Dumping database ...
Dumping PostgreSQL database gitlabhq_production ... [DONE]
2021-09-07 06:35:28 +0000 -- done
2021-09-07 06:35:28 +0000 -- Dumping repositories ...
2021-09-07 06:35:28 +0000 -- [SKIPPED]
2021-09-07 06:35:28 +0000 -- Dumping uploads ...
2021-09-07 06:35:28 +0000 -- [SKIPPED]
2021-09-07 06:35:28 +0000 -- Dumping builds ...
2021-09-07 06:35:28 +0000 -- [SKIPPED]
2021-09-07 06:35:28 +0000 -- Dumping artifacts ...
2021-09-07 06:35:28 +0000 -- [SKIPPED]
2021-09-07 06:35:28 +0000 -- Dumping pages ...
2021-09-07 06:35:28 +0000 -- [SKIPPED]
2021-09-07 06:35:28 +0000 -- Dumping lfs objects ...
2021-09-07 06:35:28 +0000 -- [SKIPPED]
2021-09-07 06:35:28 +0000 -- Dumping container registry images ...
2021-09-07 06:35:28 +0000 -- [SKIPPED]
Creating backup archive: 1630996528_2021_09_07_14.3.0-pre_gitlab_backup.tar ... done
Uploading backup archive to remote storage stanhu-s3-workhorse-testing ... rake aborted!
Excon::Error::Forbidden: Expected(200) <=> Actual(403 Forbidden)
excon.error.response
  :body          => "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>P2NMF0285BM5EQPW</RequestId><HostId>9ZqY8rdbV+sRfMVQfnd9Unqb/DpjQUv+m2OLpjByYdZfi0fVEFvq8EQcbGhMQSYRd+dBexO0BnQ=</HostId></Error>"
<snip>

After

Added this block:

gitlab_rails['backup_upload_storage_options'] = {
  'server_side_encryption' => 'aws:kms',
  'server_side_encryption_kms_key_id' => 'arn:aws:kms:REDACTED'
}
$ sudo gitlab-rake gitlab:backup:create SKIP=uploads,builds,artifacts,lfs,registry,pages,repositories
2021-09-07 06:38:29 +0000 -- Dumping database ...
Dumping PostgreSQL database gitlabhq_production ... [DONE]
2021-09-07 06:38:33 +0000 -- done
2021-09-07 06:38:33 +0000 -- Dumping repositories ...
2021-09-07 06:38:33 +0000 -- [SKIPPED]
2021-09-07 06:38:33 +0000 -- Dumping uploads ...
2021-09-07 06:38:33 +0000 -- [SKIPPED]
2021-09-07 06:38:33 +0000 -- Dumping builds ...
2021-09-07 06:38:33 +0000 -- [SKIPPED]
2021-09-07 06:38:33 +0000 -- Dumping artifacts ...
2021-09-07 06:38:33 +0000 -- [SKIPPED]
2021-09-07 06:38:33 +0000 -- Dumping pages ...
2021-09-07 06:38:33 +0000 -- [SKIPPED]
2021-09-07 06:38:33 +0000 -- Dumping lfs objects ...
2021-09-07 06:38:33 +0000 -- [SKIPPED]
2021-09-07 06:38:33 +0000 -- Dumping container registry images ...
2021-09-07 06:38:33 +0000 -- [SKIPPED]
Creating backup archive: 1630996713_2021_09_07_14.3.0-pre_gitlab_backup.tar ... done
Uploading backup archive to remote storage stanhu-s3-workhorse-testing ... done
Deleting tmp directories ... done
done
Deleting old backups ... skipping
Warning: Your gitlab.rb and gitlab-secrets.json files contain sensitive data
and are not included in this backup. You will need these files to restore a backup.
Please back them up manually.
Backup task is done.

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion

Required

  • Merge Request Title, and Description are up to date, accurate, and descriptive
  • MR targeting the appropriate branch
  • MR has a green pipeline on GitLab.com
  • Pipeline is green on dev.gitlab.org if the change is touching anything besides documentation or internal cookbooks
  • trigger-package has a green pipeline running against latest commit

Expected (please provide an explanation if not completing)

  • Test plan indicating conditions for success has been posted and passes
  • Documentation created/updated
  • Tests added
  • Integration tests added to GitLab QA
  • Equivalent MR/issue for the GitLab Chart opened
Edited by Stan Hu

Merge request reports