Switch production to use assets from object storage

Production Change - Criticality 3 C2

Change Objective	Assets for canary will be loaded from object storage
Services Impacted	GitLab.com CDN, all requests made to /assets
Change Team Members	@jarv
Change Severity	C2
Buddy check or tested in staging	The change has been validated on staging
Schedule of the change	2019-05-20 12:00UTC
Duration of the change	20 minutes
Detailed steps for the change. Each step must include:	- pre-conditions for execution of the step, - execution commands for the step, - post-execution validation for the step , - rollback of the step

Summary

This change issue changes where GitLab.com serves assets. Currently they are being served at the origin, from disk on gitlab.com. This will change our CDN and HAProxy configuration so that they are served from google object storage.

Prep

• Ensure that the assets job has run for 11.11.0-rc3.ee.0 https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/jobs/504608

• Ensure that 11.11.0-rc3.ee.0 is deployed to both production and canary https://dashboards.gitlab.net/d/llfd4b2ik/canary?orgId=1

• Verify the following production CDN asset link for validation, this should give a 200 status code

• Verify the following production CDN upload link for validation, this should give a 200 status code

• Verify the following canary CDN asset link for validation, this should give a 200 status code

• Verify the same asset can be fetched directly from object storage

• Verify the following canary CDN upload link for validation, this should give a 200 status code

Canary

Before applying the new configuration run the following two commands in a terminal to ensure that assets are returning 200s. This should be run through the entire process, if at any time we see anything but a 200 we should abort the procedure and rollback.

while true; do \
  curl -s -w "%{http_code} " \
  https://gl-canary.freetls.fastly.net/assets/application-39b154eeefef684cb6d56db45d315f8e9bf1b2cc86cf24d8131c674521f5b514.css \
 -o /dev/null; \
 done

while true; do \
  curl -s -w "%{http_code} " \
  https://gl-canary.freetls.fastly.net/uploads/-/system/user/avatar/1254817/avatar.png?width=23 \
 -o /dev/null; \
 done

Steps

• Create a new origin host in fastly
  • In the Address field, type storage.googleapis.com.
  • In the comment field type gprd assets
  • In the Transport Layer Security (TLS) area, ensure that Enable TLS? is set to Yes
  • In the Transport Layer Security (TLS) area, ensure that storage.googleapis.com in the Certificate hostname field.
  • From the Shielding menu, select an interconnect location from the list of shielding locations, we will use Ashburn DCA
  • Attach the condition req.url ~ "^/assets/" to the new asset origin
  • Attach the condition req.url !~ "^/assets/" to the existing origin
• In settings -> override host add gitlab-gprd-assets.storage.googleapis.com
• In domains create gitlab-gprd-assets.storage.googleapis.com, this is necessary for shielding to work properly
• Inspect the diff before applying

Production

Before applying the new configuration run the following two commands in a terminal to ensure that assets are returning 200s. This should be run through the entire process, if at any time we see anything but a 200 we should abort the procedure and rollback.

while true; do \
  curl -s -w "%{http_code} " \
  https://assets.gitlab-static.net/assets/application-39b154eeefef684cb6d56db45d315f8e9bf1b2cc86cf24d8131c674521f5b514.css \
 -o /dev/null; \
 done

while true; do \
  curl -s -w "%{http_code} " \
  https://assets.gitlab-static.net/uploads/-/system/group/avatar/6543/logo-extra-whitespace.png?width=15 \
 -o /dev/null; \
 done

• Repeat the same exact steps for canary, but for the production CDN https://manage.fastly.com/configure/services/69F26noirwWBeIYllOadQm/versions/11/domains
• Enable the asset proxy, this will proxy all /assets requests from haproxy directly to object storage https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/1122

Rollback

Roll back by switching to the previous config version:

• Canary should be rolled back to Version 20
• Production should be rolled back to Version 11 https://manage.fastly.com/configure/services/69F26noirwWBeIYllOadQm/versions/11/origins