2018-05-17: staging failover test plan execution

Test Plan

High-level Overview

This test plan requires some manual testing as well as some automated testing using GitLab-QA.

The automated and manual QA processes can be conducted in parallel with one another.

The plan will also involve testing during two distinct phases of the failover plan:

  • During the Blackout (inside the maintenance window period): all the tests conducted during this period are related to features critical to the operation of GitLab.com. The maintenance window cannot be closed until these tests pass.
  • After the Blackout (after the maintenance window period): these tests are non-critical, but, in order to keep the maintenance window as short as possible, should be conducted after the new failover GitLab instance is public.

Automated QA

During the Blackout

@meks to run GitLab-QA against failed-over environment.

  • GitLab-QA complete with 100% pass

Functional tests @meks 

env GITLAB_USERNAME=gitlab-qa GITLAB_PASSWORD=... gitlab-qa Test::Instance::Any EE latest https://staging.gitlab.com

Results

Finished in 12 minutes 43 seconds (files took 0.33427 seconds to load)
19 examples, 1 failure

Failed examples:

rspec ./qa/specs/features/merge_request/create_spec.rb:3 # creates a merge request user creates a new merge request

Randomized with seed 29203

Manual QA

During the Blackout

  1. Email (PRODUCTION ONLY)
  2. Does outgoing notification continue to work from new sending hosts DKIM, SPF - @toon
  3. Do incoming emails update notes - @toon
  4. Repository Operations
  5. pushing to protected branch (provide access when it should and protect when it should) - @mkozono
  6. ssh - @mkozono
  7. forking (failed so far) - @mkozono
  8. Access existing LFS object - @mkozono
  9. Create new LFS object - @mkozono
  10. Uploads
  11. Access existing upload - @mkozono
  12. Create new upload - @mkozono
  13. Artifacts
  14. Access existing artifacts (failed) - @mkozono
  15. Create new artifacts - @mkozono
  16. GitLab Service Desk (PRODUCTION ONLY)
  17. incoming emails - @stanhu
  18. AlertManager Alerts
  19. AlertManager - @dawsmith on behalf of Production Team

After the Blackout

  1. Wiki Operations
  2. Access existing wiki - @mkozono
  3. Create new wiki - @mkozono
  4. Pages
  5. Access existing page - @toon
  6. Create new page - @toon
  7. Access from Azure (need a browser on a VM in Azure ?) - @toon
  8. Access from GCP (need a browser on a VM in GCP ?) - @toon
  9. Mirror
  10. Push - @vsizov
  11. Pull - @vsizov
  12. Backups
  13. Redis - @dawsmith on behalf of Production Team
  14. Postgres - @dawsmith on behalf of Production Team
  15. Git Data - @dawsmith on behalf of Production Team
Edited by Mek Stittri