Support AWS SSE-KMS in backups

Merged Stan Hu requested to merge sh-backup-support-aws-kms into master

What does this MR do?

As described in https://docs.aws.amazon.com/AmazonS3/latest/userguide/serv-side-encryption.html, AWS supports three different modes for encrypting S3 data:

  1. Server-Side Encryption with Amazon S3-Managed Keys (SSE-S3)
  2. Server-Side Encryption with Customer Master Keys (CMKs) Stored in AWS Key Management Service (SSE-KMS)
  3. Server-Side Encryption with Customer-Provided Keys (SSE-C)

Previously, SSE-S3 and SSE-C were supported via the backup.upload.encryption and backup.upload.encryption_key configuration options.

SSE-KMS was previously not supported in backups because there was no way to specify which customer-managed key to use. However, we did support SSE-KMS with consolidated object storage enabled for other CI artifacts, attachments, LFS, etc. Note that SSE-C is NOT supported here.

In consolidated object storage, the storage_options Hash provides the server_side_encryption and server_side_encryption_kms_key_id parameters that allow admins to configure SSE-KMS. We reuse this configuration in backups to support SSE-KMS.

Relates to #338764 (closed), #328763

Testing

  1. Configure gitlab.yml with AWS credentials:
  backup:
    path: "tmp/backups"   # Relative paths are relative to Rails.root (default: tmp/backups/)
    gitaly_backup_path: /Users/stanhu/gitlab/gdk-ee/gitaly/_build/bin/gitaly-backup # Path of the gitaly-backup binary (default: searches $PATH)
    # archive_permissions: 0640 # Permissions for the resulting backup.tar file (default: 0600)
    # keep_time: 604800   # default: 0 (forever) (in seconds)
    # pg_schema: public     # default: nil, it means that all schemas will be backed up
    upload:
      remote_directory: stanhu-s3-workhorse-testing
      connection:
        provider: AWS
        region: us-west-2
        aws_access_key_id: REDACTED
        aws_secret_access_key: REDACTED
      storage_options:
        server_side_encryption: 'aws:kms'
        server_side_encryption_kms_key_id: 'arn:aws:kms:us-west-2:REDACTED'
  1. Define an S3 bucket policy that rejects uploads that are missing server side encryption headers:
{
    "Version": "2012-10-17",
    "Id": "PutObjPolicy",
    "Statement": [
        {
            "Sid": "DenyIncorrectEncryptionHeader",
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::stanhu-s3-workhorse-testing/*",
            "Condition": {
                "StringNotEquals": {
                    "s3:x-amz-server-side-encryption": "aws:kms"
                }
            }
        },
        {
            "Sid": "DenyUnEncryptedObjectUploads",
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::stanhu-s3-workhorse-testing/*",
            "Condition": {
                "Null": {
                    "s3:x-amz-server-side-encryption": "true"
                }
            }
        }
    ]
}
  1. Run the backup:
bundle exec rake gitlab:backup:create SKIP=uploads,builds,artifacts,lfs,registry,pages,repositories
  1. Then check the S3 bucket for the uploaded file:

image

Edited by Stan Hu