Enable CloudFront on packages.gitlab.com

Production Change - Criticality 2 C2

Change Objective Use CloudFront with PackageCloud. This is a required change before 2019-06-24 or everything will stop working.
Change Type C2 configuration update
Services Impacted packages.gitlab.com
Change Team Members @ahanselka
Change Severity Must be done before 2019-06-24
Buddy check A colleague will review the change
Tested in staging No staging for packages
Schedule of the change 2019-06-13 23:00 UTC
Duration of the change TBD

Change Steps

  • Ensure the AWS key that we use with PackageCloud has relevant permissions to manage CloudFront distributions. The required permissions are:
    • Create an IAM role and policy.
    • Create a KMS key and policy.
    • Create/Modify Lambda Functions.
    • Create CloudFront Origin Access Identity.
    • Attach Bucket Policy to S3 bucket.
    • Create/Modify CloudFront Distributions.
  • Run packagecloud-ctl reindex-everything
  • Disable chef-client: service chef-client stop
  • Run packagecloud-ctl generate-cloudfront-secret
  • Add generated secret to packagecloud.rb manually
  • Run packagecloud-ctl bootstrap-cloudfront
  • Run the command that the bootstrap-cloudfront generates
  • Add the returned values and the generated secret above to chef so that the packagecloud.rb configuration can be updated
  • packagecloud-ctl reconfigure
  • packagecloud-ctl restart

Rollback

We can easily roll back by setting packagecloud_rails['cloudfront_enabled'] to false. This will be a configurable setting in chef once gitlab-cookbooks/gitlab-packagecloud!16 (merged) is merged.

Validation

Check headers before and after:

curl -I https://packages.gitlab.com/gitlab/gitlab-ce/packages/debian/jessie/gitlab-ce_11.11.3-ce.0_amd64.deb/download.deb
HTTP/1.1 302 Found
Server: nginx
Date: Thu, 13 Jun 2019 16:09:26 GMT
Content-Type: text/html;charset=utf-8
Content-Length: 0
Connection: keep-alive
Location: https://packages-gitlab-com.s3-accelerate.amazonaws.com/7/8/debian/package_files/52621.deb?AWSAccessKeyId=AKIAJ74R7IHMTQVGFCEA&Signature=Kq8sv6P66WBEdhlZCGBaJjfYekY%3D&Expires=1560442468

Docs

Edited by Alex Hanselka