Fulfillment QE Test task - improve E2E tests efficiency
Problem
Related to OKR - https://gitlab.com/gitlab-com/gitlab-OKRs/-/work_items/3550+. This issue is to identify Fulfillment QE action items to improve E2E tests efficiency.
We previously had UAT testing sessions for Project Matterhorn, which included various downstream teams to perform manual verification for multiple test scenarios. It required lots of coordinations between teams and the all manual test artifacts generation (multiple times) was tedious and time consuming. We want to come up with some action items to help reduce the manual efforts required for e2e testing, and possibly draft out an e2e testing plan for each team when it comes to manual testing.
Prerequisite
We'd like to understand first for each team -
- What are missing from the current automated e2e tests, e.g. critical flows, missing integrations like SFDC etc?
- How often do we plan on conducting UAT testing sessions? How do we define confidence level when manual testing sessions or downstream team are not included (or can we quantify an automated test coverage to make the call)
- Due to the nature of current automated e2e tests (purely blackbox in live environment, no access to rails console etc.), for more complex scenarios, we probably need more APIs to support testing. Could it be prioritized?
Result
- Identify current test gap in automated e2e tests, and things/verifications can be added.
- Improve tooling/pipeline that could save some manual work when manual testing is required.
- Improve labeling and documentation for Testcase Management, and make it the SSoT for all automated tests.
- Create a manual UAT testing plan for each team
Next steps (if any)
How will we measure success?
Edited by Chloe Liu