Skip to content

Add support to AWS encryption with KMS key + S3 endpoint URL

Gregory OpenIT requested to merge (removed):toolbox-encryption-regional-patch into master

Context

STS regionalized endpoint and AWS service endpoint

When using AWS in an air-gapped environment, global STS and S3 endpoints are not accessible. Instead, STS regionalized endpoint and AWS service endpoint must be used.

When using a STS regionalized endpoint, the region has to be specified. But my tests revealed that the environment variable AWS_REGION is ignored by aws cli during the aws s3api head-bucket --bucket call, the STS endpoint remains the global https://sts.amazonaws.com. If the option --region is specified, the STS endpoint is correctly adapted to sts..amazonaws.com. This looks like a bug of the aws cli but for now it is simpler to enforce the usage of this option.

S3 encryption with customer key

If the S3 bucket is encrypted with a customer key in KMS, this key must be passed to aws cli.

What does this MR do?

The toolbox does not currently support:

  • AWS S3 encryption with a customer KMS key,
  • AWS S3 endpoint URL
  • AWS STS regionalized endpoint

This MR brings 3 new arguments to gitlab-toolbox/scripts/bin/backup-utility:

     --aws-kms-key-id                       XXX 
     --aws-s3-endpoint-url                  XXX
     --aws-region                           XXX

These arguments are also forwarded to the ruby library gitlab-toolbox/scripts/lib/object_storage_backup.rb with two new arguments aws_s3_settingsand aws_kms_settings.

Test

In the helm chart values, the extraArgs can now be completed with these new options, such as

 toolbox:
    backups:
      cron:
        extraArgs: --aws-kms-key-id <KEY_ID> --aws-s3-endpoint-url <SPECIFIC_URL>

Checklist

Required

  • Merge Request Title, and Description are up to date, accurate, and descriptive
  • MR targeting the appropriate branch
  • MR has a green pipeline on GitLab.com

Expected (please provide an explanation if not completing)

  • Test plan indicating conditions for success has been posted and passes
  • Documentation created/updated
  • Integration tests added to GitLab QA
  • The impact any change in container size has should be evaluated

No new functions are added. This MR only implements more arguments for the AWS cli commands. Does it require test plan and integration test ? If yes, could someone help me on it ?

Edited by Mitchell Nielsen

Merge request reports