2018-05-17: staging failover test plan execution
Test Plan
High-level Overview
This test plan requires some manual testing as well as some automated testing using GitLab-QA.
The automated and manual QA processes can be conducted in parallel with one another.
The plan will also involve testing during two distinct phases of the failover plan:
- During the Blackout (inside the maintenance window period): all the tests conducted during this period are related to features critical to the operation of GitLab.com. The maintenance window cannot be closed until these tests pass.
- After the Blackout (after the maintenance window period): these tests are non-critical, but, in order to keep the maintenance window as short as possible, should be conducted after the new failover GitLab instance is public.
Automated QA
During the Blackout
@meks to run GitLab-QA against failed-over environment.
-
GitLab-QA complete with 100% pass
@meks
Functional testsenv GITLAB_USERNAME=gitlab-qa GITLAB_PASSWORD=... gitlab-qa Test::Instance::Any EE latest https://staging.gitlab.com
Results
Finished in 12 minutes 43 seconds (files took 0.33427 seconds to load)
19 examples, 1 failure
Failed examples:
rspec ./qa/specs/features/merge_request/create_spec.rb:3 # creates a merge request user creates a new merge request
Randomized with seed 29203
Manual QA
During the Blackout
- Email (PRODUCTION ONLY)
-
Does outgoing notification continue to work from new sending hosts DKIM, SPF - @toon -
Do incoming emails update notes - @toon - Repository Operations
-
pushing to protected branch (provide access when it should and protect when it should) - @mkozono -
ssh - @mkozono -
forking (failed so far) - @mkozono -
Access existing LFS object - @mkozono -
Create new LFS object - @mkozono - Uploads
-
Access existing upload - @mkozono -
Create new upload - @mkozono - Artifacts
-
Access existing artifacts (failed) - @mkozono -
Create new artifacts - @mkozono - GitLab Service Desk (PRODUCTION ONLY)
-
incoming emails - @stanhu - AlertManager Alerts
-
AlertManager - @dawsmith on behalf of Production Team
After the Blackout
- Wiki Operations
-
Access existing wiki - @mkozono -
Create new wiki - @mkozono - Pages
-
Access existing page - @toon -
Create new page - @toon -
Access from Azure (need a browser on a VM in Azure ?) - @toon -
Access from GCP (need a browser on a VM in GCP ?) - @toon - Mirror
-
Push - @vsizov -
Pull - @vsizov - Backups
-
Redis - @dawsmith on behalf of Production Team -
Postgres - @dawsmith on behalf of Production Team -
Git Data - @dawsmith on behalf of Production Team
Edited by Mek Stittri