Test Plan for Maintenance Mode on Staging

Scheduled for Monday, January 25 @ 11:30 UTC

This issue is for defining scope and planning execution of maintenance mode testing on staging.

  • What will be tested
  • How they will be tested (automated UI, automated API, manual)
  • How to execute automated tests
  • How results will be recorded

Setting up data

1/18 UPDATE: Projects to be used for staging tests are found in https://staging.gitlab.com/jsl-qa-sandbox-group/maintenance-mode. Nick and Aakriti have been made maintainers for that group.

We need to set up projects, issues, etc. on staging before turning maintenance mode on.

One idea is to keep all projects created during a full QA test suite run (making sure group/subgroups are not deleted after test runs), since these generate data for all the different features we currently test.

Or we could use generate_perf_testdata QA rake task to generate projects, then create additional resources as needed, and then export these projects to staging. Certain items like job artifacts and CI Variables cannot be exported, so those would be created on staging after importing the project(s).

We could use resource fabricators from QA tests to create pipelines and other resources that the rake task does not generate.

Automated Tests via API (list in progress)

Tests run against the API can be managed using Postman.

To do: check how test results are recorded with Postman

Manual tests (list in progress)

Create a list of manual test with relevant URLs. We'll record the outcome of the tests there as well.

Link to spreadsheet: https://docs.google.com/spreadsheets/d/1bhkTBLY_BTw3Zh-2Wl-nMNYBGlSKS9eSR8rIDAxqvVs/edit#gid=486060733

  • git push via SSH
  • Merge an MR
  • git push to Geo secondary via SSH
  • git push to Geo secondary via HTTP
  • quick actions on issues, MRs
  • email?
  • search
  • boards
  • web ide

Automated Tests via UI

These ended up being too brittle as automated tests because of the data required, and not worth the time to troubleshoot since automated pipeline tests will generate data via API (before enabling maintenance mode). I have replaced these with API and additional manual tests.

Edited by Jennifer Louie