Backup Appreciation Day 2017-08-31

After switching to the new primary database, it means that...

It's that time again, backup appreciation day! The goal of this day is to ensure our backups are in fact actually backing up. There are several things to test here.

• Restore WAL-E backup for our production DB (relevant documentation)
• Restore of a secondary servers with WAL-E w/GPG encryption (relevant documentation):
• dev.gitlab.com
• version.gitlab.com
• Test restore of an Azure snapshot (relevant documentation)
• Test restore of PackageCloud database (relevant documentation)

Automate steps that can be automated. Previous B.A.D.: https://gitlab.com/gitlab-com/infrastructure/issues/2414

Related: https://gitlab.com/gitlab-com/infrastructure/issues/2665 -- the prod db box should be handed over to @yorickpeterse for experiments.