How can we better achieve community MR review SLO times
Description
Hi GitLab Team members from all around our precious
There is some contrarian data on our SLO's (https://docs.gitlab.com/ee/development/code_review.html#review-response-slo which speaks of 2 business days, and https://about.gitlab.com/handbook/engineering/quality/merge-request-triage/#wider-community-merge-request-triage-slos which speak of 7 days to review.) I took the 7 business days to be lenient but even then we see that many groups/stages don't meet this criteria.
I am looking for experiments/actions/thoughts on this topic to then take action that can lead us towards achieving the SLO. It would be great to have your input.
Current assumptions or thoughts I have:
- Should we add more ICs in these stages for the specific domains that are struggling to keep up?
- Should we re-assess our priorities in those stages that are not meeting the SLO of 7 days and focus less on our own product roadmap & get more focus on community contributions?
Relevant Slack discussion: https://gitlab.slack.com/archives/C2T9APP9C/p1658231769664109
Stages with open MRs that are ready for review versus time since it was in ready for review
Current open MRs: 89
Current total average: 20 days
Targeted SLO: 7 days
| Stage | Open MRs | Average time the MR is in the Ready For Review state |
|---|---|---|
| Verify | 39 | 28 days |
| Create | 14 | 10 days |
| Manage | 13 | 11 days |
| Ecosystem | 5 | 15 days |
| Plan | 4 | 15 days |
| Package | 2 | 15 days |
| Secure | 2 | 8 days |
Note: Data can be found in realtime at https://app.periscopedata.com/app/gitlab/729542/Wider-Community-PIs.
I'll fill in this section with a summary based on people's feedback on what we can do to improve this.
Stop
Start
- track these community merge request SLOs by project rather than by group or stage
- Out of the total number of maintainer reviews requested (e.g. ~4,200 in the past 30 days), what percentage of these are for community contributions?
- For each project, what is the average number of reviews (daily or monthly) for each maintainer? What percentage of them are community contributions? High average review count would indicate a capacity problem, but if the engineers have a manageable review count then the problem lies elsewhere (ex: processes)
- Of the community contribution MRs which have gone stale, what stage of the review process they are stuck on? (initial review pending, maintainer review pending)
- educate more people on the workflow labels and enforce & remind GitLab team members when appropriate
- Track average & median time an MR is in the review state, so if an MR has seen 3 transitions from ~"workflow::In dev" to workflowready for review and back. We should take all the time this MR was in ready for review into account. (5 days in review, 7 days in review and 20 days in review gives us an average of 10,6, while today we only take the 20 days into account)
- Track time to first review in accordance of our SLO target (https://docs.gitlab.com/ee/development/code_review.html#review-response-slo)
- Define time to next review as an SLO and measure this
- Make it explicit that our SLO includes community contributions here: gitlab-org/gitlab!93269 (closed)
- Assess whether we should stick to using workflow labels for the review process vs using our own labels so that this workflow can work independently from the product process, which could potentially use workflow labels differently.
- Train or make it more obvious to wider community members that they can set all the allowed workflow labels such as workflowblocked , ~"workflow::In dev" & ~"workflow::In review". Existing command:
@gitlab-bot label ~"workflow::in dev" - Consider adding a workflow label : "workflow::reviewed & tested by the community"
- How many are being reviewed/merged per project / per group so we can set a baseline & relay this information to Mek & Christopher so they can set a goal/directions
Try
Reduce current backlog & temporarily make a push in the groups/stages that don't meet the SLO to reduce the current backlog, after assessing the MRs that we should push on.
- First priority should go to MRs that are fixing open bugs and don't require a lot of work to be mergeable (e.g. https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests?scope=all&state=opened&label_name[]=Community%20contribution&label_name[]=type%3A%3Abug)
- Then MRs that are enhancing existing features and don't require a lot of work to be mergeable (e.g. https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests?scope=all&state=opened&label_name[]=Community%20contribution&label_name[]=type%3A%3Afeature)
- And finally, MRs that are maintenance-related and don't require a lot of work to be mergeable (e.g. https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests?scope=all&state=opened&label_name[]=Community%20contribution&label_name[]=type%3A%3Amaintenance)