FY21-Q1 Dev QE OKR: Create a reliable suite of end-to-end tests to guard every deploy in the release process and gain insights into historical reliability => 92%
Key Results
-
Key result: Create reliable full end-to-end suite definition to run with every deploy in the release process. => 90%, Process is defined, next iteration in documentation yet to be done
- Create the definition of a
reliable
end-to-end test suite https://gitlab.com/gitlab-org/quality/team-tasks/issues/337 - Build a process to promote test into this reliable test suite - https://gitlab.com/gitlab-org/quality/team-tasks/issues/378 and #6678 (closed)
- Run all
reliable
end-to-end tests as part of the release process gitlab-com/gl-infra/delivery#677 (closed) and https://gitlab.com/gitlab-org/quality/team-tasks/issues/423
- Create the definition of a
-
Improve reporting of test results in slack channels that is part of the release process, qa-staging, qa-production to have stage specific result summary. https://gitlab.com/gitlab-org/quality/team-tasks/issues/372 => 100%
-
Identify historical pass rate of each end-to-end test in staging for reliability https://gitlab.com/gitlab-org/quality/team-tasks/issues/406 => 85%, APIs for getting JUnit JSON Object in progress, Creating a Testcases Dashboard is yet to start
- Dependencies gitlab-org/gitlab#202525 (closed)
CC @at.ramya @gl-quality/dev-qe
Retrospection
Good
- Reliable tests process is useful and has helped bring clarity into the release process.
- Slack improvements to the reporting is very useful.
Bad
- We still do not have insights into historical tests reliability.
- We did not make progress towards the end of the quarter due to higher priority work streams.
- Mark -> Gitaly HA tasks and few enhancements related to Testcase Management System & Customer Escalation tasks
- Tomislav -> Gitaly HA tasks
- Sanad, Desiree -> Githost working Group
Try
- We need to add more reliable tests that has a high impact value of detection on features that are used most often.
Edited by Mek Stittri