FY20-Q2 Quality Department OKR: Increase Engineering Productivity => 87%
Improve test engineering process, reduce test execution time, reduce review app deploy time, make tests easier to contribute, make test failure debugging easier for developers.
-
Key Result Dev
: Reduce end-to-end tests runtime in review apps by 50% and makereview-qa-all
to be anmandatory joboptional job(allowed to fail) in CI gitlab-org&1276=> 95%
-
Key Result Dev
: Reduce end-to-end tests runtime in package-and-qa job by 50% https://gitlab.com/gitlab-org/gitlab-ce/issues/64953=> 90%
-
Key Result Dev
: Roll out training program teaching how to write end-to-end tests, record and publish training sessions. gitlab-org&1183 (closed)=> 100%
-
Key Result Ops & CI/CD
: Improve summary test report for end-to-end tests. https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/28834=> 90%
-
Key Result Secure
: Complete end-to-end tests for secure. gitlab-org/quality/testcases#7 (closed)=> 83%
-
Key Result Engineering Productivity
: Increase review app deployment success rate to 90%. https://gitlab.com/gitlab-org/quality/team-tasks/issues/93=> 50%
-
Key Result Engineering Productivity
: Make GitLab Insights generally available. https://gitlab.com/gitlab-org/quality/team-tasks/issues/137=> 100%
-
Key Result Secure
: Add seed data for secure in review apps. https://gitlab.com/gitlab-org/gitlab-ee/issues/8466=> 0%
Retrospective
Good
- Increased collaboration in the department and pair programing when we implemented the HTML report.
- Running tests in parallel was well planned with cost analysis.
- We completed the 'training program' OKR in two months.
Bad
- We didn't fully complete 5 out of 8 KRs
- Many dependencies for review apps tied to us delivering success rate.
-
review-qa-all
seems blocked on review app success rate, we may need to fall back to package-and-qa as a working quality gate. - Attempting to run
package-and-qa
in parallel highlighted how a few long jobs slow the whole suite down and limit the benefits of parallelization.
Try
- Better planning, consider sync planning. E.g. a one-off meeting to quickly get input in a google doc then create issues. We try our best to do async communication, however planning async is harder we should try to get planning done 1-2 weeks into the quarter.
- Consider integration tests or any detection in review app dependency projects. Catch issues earlier before we consume them in review apps.
- A more comprehensive training program. AMA was great but most questions were very high level. It shows that development hasn't reached or consumed the more advanced concepts of the QA framework yet. e.g PageObject structure, readability and etc.
- Try adding weights to tasks to achieve Key Result.
- Break the OKR into monthly deliverables and align them to monthly milestones (%12.2, etc.)
Edited by Ramya Authappan