FY20-Q4 Quality Product OKR: Increase Engineering efficiency and productivity by speeding up test and pipeline duration => 77%

• Key result: Increase GitLab review app success rate from 29% (October 2019) to 70% => 100%
• Key result: Reduce regression suite run time in the release process by 20% from 70 mins to 56 mins https://gitlab.com/gitlab-org/quality/team-tasks/issues/284 => 40% Currently at ~60 mins.
  • We however made progress with the end-to-end test stability, we are down to only 4 flaky tests.
• Key result: Provide stage specific reporting mechanism for tests that run as part of the release process https://gitlab.com/gitlab-org/quality/team-tasks/issues/316 => 90%, change in direction and dogfooding our built in junit report feature.

Retrospective

GOOD

• Review app stability is better.
• Master stability is much better.
• We dogfooded the test reporting feature.

BAD

• We worked on master stability but it wasn't captured as part of the OKR.
• We shifted direction mid quarter but OKR and issues could be made clear that we are dogfooding test_report
• No one took up the reporting and regression reduction time when the original engineers were on vacation.

TRY

• Better planning and coordination between internal teams.
• We need more analysis on what tests are taking the longest.
• Start using weights so capacity is clear.