FY22-Q1 KR: Improve quality SLOs for S1s and S2s for Code Review and Source Code => 61%

In the past year we have seen a steady increase in backlog bugs for both Code Review and Source Code. As key parts of our product we would like to see our Sev1 and Sev2 bugs under a more SLA based solution. In this quarter we would like to see improved SLOs.

Metrics being analyzed:

It's important to note that while SLOS are showing Mean Time to Resolve being below target, our open bug count is going up. We think this may be that once bugs are triaged they are either fixed or ignored from a priority consideration.

Steps to address this may include

  • Make sure data is correct for these groups and update to reflect proper information
  • Consider changing MTTR to be (open bug age + closed bug aged)/(# open bugs + # closed bugs)
  • Review current open bugs for how long they were open
  • Work with product to demonstrate this problem and get bugs prioritized to bring down so that open bugs more closely matches a months worth of opened bugs

Good

Things that went well. If your team accomplished your goal, what contributed to the team's success?

  • Code Review: This OKR was really insightful and prompted us to look into our processes. The result of this OKR was a new process in place for the future. We also cleaned out a lot of issues since the quarter began, mostly through better triaging, proactive investigations (looking at the issues as they came in), and refinement of old issues (some issues were duplicates or already resolved).
  • Code Review: Our mean-time-to-close for S2's increased throughout the quarter (bad) while old issues got closed, but our number of closed issues and the percentage closed below SLO also increased (good). This tells me that if we continue on the same path with the new process, our mean-time-to-close will begin dropping.

Bad

Things that did not go well, but are generally one-off events—things that we don’t expect to repeat. If your team didn’t accomplish your goal, what obstacles were encountered?

  • Code Review: Even though we closed 15 issues that had missed their SLO's + worked on issues that had not yet missed their SLO, 10 new issues missed their SLO throughout the quarter. This almost cancelled out our progress with the OKR.

Do Better

Things that we can do better next time. These can include a suggestion on how to do it better.

  • Code Review: Avoid setting goals that are moving targets (new issues coming in) and use concrete numbers instead when applicable. The intent behind the goal, which was to reduce the open bug count to be more manageable, become more proactive, and decrease the age, got lost when we needed to account for the new issues coming in.

cc: @meks

Edited by Christopher Lefelhocz