Test plan to verify failed-over instance

As part of the GCP Migration, we need a comprehensive test plan that we can use to certify that the failed over instance of GitLab is working correctly.

This plan should be used in Staging and Production Environments

Some questions:

• Can we use GitLab QA against the failed-over environment?
• How much functionality that is not included in GitLab QA will we need to test manually?
• Once the production environment is failed over to GCP..
  • What's the best way for us get access to the instance, for testing, without allowing the public on yet?
  • How long will the QA of the failed-over production environment? This will affect the overall outage window time.

---

Test plan

We have a number of functionality covered with test automation. We aim to run these automated tests as soon as the failover has finished.

High level plan

Staging

1. Run the automated tests on staging right after failover

• Triage test results
• Rerun tests / debug & fix as necessary

1. Start the manual testing per areas discussed below per assignments

• Track this in a staging manual test issue, each owner is responsible to update status and check this off

Production

1. Run the automated tests on staging right after failover

• Triage test results
• If we do this right we should't run in the the same failures we fix in staging.

1. Start the manual testing per areas discussed below

• Track this in a staging manual test issue, each owner is responsible to update status and check this off
• Communicate with RMs

During the testing phase, we should have assigned production engineers to be ready to fix issues in production.

Automated Tests

These are the automated features:

• Login to the site
• Basic features
• CI/CD pipeline
  • pushing code
  • registering runners
  • using variables
  • deploy tokens
• Repository operations
  • https
• Traces
• CI runners
  • receiving jobs
  • uploading traces and artifacts Run automated tests on staging after the failover

Run command

env GITLAB_USERNAME=gitlab-qa GITLAB_PASSWORD=... gitlab-qa Test::Instance::Any EE latest https://staging.gitlab.com

Geo tests

CHROME_HEADLESS=0 GITLAB_USERNAME=gitlab-qa GITLAB_SANDBOX_NAME=mkozono-gitlab-qa-sandbox bin/qa QA::EE::Scenario::Test::Geo --primary-address https://gitlab.com --secondary-address https://gprd.gitlab.com --primary-name primary --secondary-name secondary --without-setup

Manual Testing

Here are the area and functionality and assignments

Area	Functionality	Run in parallel w/ GitLab QA	Test on staging	Test on production	owner
Emails	-	-	-	-	@toon
	Outgoing notification does DKIM, SPF continue to work from new sending hosts, etc?)		yes	yes
	Incoming => notes		yes	yes
Repository operations	-	-	-	-	@mkozono
	pushing to protected branch (provide access when it should and protect when it should)		yes	yes
	ssh		yes	yes
	forking		yes	yes
	Access existing LFS object		yes	yes
	Push commit with LFS object		yes	yes
Wikis	-	-	-	-	@mkozono
	Access existing wiki		yes	yes
	Create new wiki		yes	yes
Uploads	-	-	-	-	@mkozono
	Access existing upload		yes	yes
	Create new upload		yes	yes
Artifacts	-	-	-	-	@mkozono
	Access existing artifact		yes	yes
	Create new artifact		yes	yes
Pages	-	-	-	-	@toon
	Access existing page		yes	post deploy
	Create new page		yes	post deploy
	Access from Azure (need a browser on a VM in Azure ?)		yes	post deploy
	Access from GCP (need a browser on a VM in GCP ?)		yes	post deploy
Mirror	-	-	-	-	@vsizov
	Push		yes	post deploy
	Pull		yes	post deploy
Backups Production team	-	-	-	-	@dawsmith
	Redis		yes	post deploy
	Postgres		yes	post deploy
	Git Data		yes	post deploy
GitLab Service Desk	-	-	-	-	@stanhu
	Do we only need to test this on production		???	yes
version.gitlab.com	-	-	-	-	@dbalexandre
	Do we only need to test this on production		???	yes
Alerts			yes	yes	@dawsmith

cc @meks @bjgopinath @jarv @grzesiek