Weekly Reliability (SRE) Team Newsletter – On-call Period:2020-12-29 - 2021-01-05
This issue has been moved and the description cleared of content to avoid polluting the search results of this tracker, see the moved issue link for the original newsletter
Activity
added Reliability-Team-Newsletter label
marked this issue as related to #12212 (moved)
There was a slack thread with a brief discussion of what I thought was an excellent suggestion - the async on-call handover has been working incredibly well, and we have a gap in our longer-term weekly on-call handovers, that would probably be filled just as effectively with an async process, as well.
Whether we continue to hold the weekly on-call handover meetings we do have, make them (more) optional, or cancel them entirely could come up for discussion at that point, as well.
-
It was pointed out in the call that this newsletter and incorporating the On-call report was intended to serve a similar purpose, and that incorporating the summary here with the on-call report would be most efficient/beneficial.
I'll start a separate discussion issueto bounce the idea around and flesh out the process/template we want to try (or not, if there's a compelling argument).Edit: I did not get back to creating a separate issue, and discussion continued in this thread. Let's keep it here for simplicity, rather than trying to split the conversation out into a separate issue unnecessarily.
Edited by Craig Barrett 
Having participated in that thread, here's my $0.02.
This newsletter and meeting serves 2 purposes: on-call incident recap, and discussing the general news with the managers.
IMO, personally I don't get a lot of value from the first purpose (on-call incident recap). It overlaps too strongly with the formal sync review meeting on the same day for interesting incidents.
Having a forum to discuss the news with the managers is nice, but when I put on my remote, distributed hat and try to be biased towards canceling sync meetings, IMO this might not be impactful enough to keep. Instead, what if we commented on this newsletter, with threads prompting zoom meetings on the occasion that they deserve them? That might also be more inclusive to anyone outside of timezones this meeting is reasonable for.
Finally, returning to @craig's original point about providing week-length context between shifts: I totally agree we need that, and suggested as much on Slack - but I do not think this meeting's on-call recap aspect serves this purpose. I'd suggest a weekly issue, similar to the every-shift issues, that all 3 on-calls can populate with longer term context to hand over to the next shift.
It overlaps too strongly with the formal sync review meeting on the same day for interesting incidents.
Additional point made about combing through the list and separating out issues that will require review. We can do this using label filters.
@cmcfarland made a good suggestion:
"Let's have the incoming on-call run the meeting, asking the relevant questions, rather than the outgoing on-call summarizing the week in it's entirety.
-
I feel like the handover meeting offers the ability to ask questions about incidents and is more likely to get a response than in a recap issue.
True, but an async recap has other benefits, besides; it can help frame that discussion, keeps the summary handover information available async, helps outgoing EOCs focus their thoughts and prioritize any discussions, and if incorporated with the weekly review issues, can serve as a reference point for further issue/MR relations to tie together threads. Altogether, it would provide a better resource overall, I think.
All that said, the proposal is not necessarily to get rid of the handover meeting, merely to extend our current async handover process to include a final weekly summary, and see how that impacts the content or facilitation of the weekly handover meeting.
This newsletter and meeting serves 2 purposes: on-call incident recap, and discussing the general news with the managers.
IMO, personally I don't get a lot of value from the first purpose (on-call incident recap). It overlaps too strongly with the formal sync review meeting on the same day for interesting incidents.
The on-call report that gets added to these issues is great, but it's simply a dump of raw statistics. The weekly summary gives us a chance to engage more fully with the content to discuss longer-term ongoing issues, add nuanced context around past incidents, discuss ad-hoc process adjustments, silences, etc.
I feel like the handover meeting offers the ability to ask questions about incidents and is more likely to get a response than in a recap issue.
Personally, I feel like information recorded in issues is much more accessible async. I know we can record meetings and update meeting agendas in Google docs, but I find myself leaning on gitlab-centric sources of information far more than anything else. In that regard, this would provide a much more useful and durable method for storing and sharing information.
Having a forum to discuss the news with the managers is nice, but when I put on my remote, distributed hat and try to be biased towards canceling sync meetings, IMO this might not be impactful enough to keep. Instead, what if we commented on this newsletter, with threads prompting zoom meetings on the occasion that they deserve them? That might also be more inclusive to anyone outside of timezones this meeting is reasonable for.
I think it comes down to making a small iteration, and setting a timeframe to evaluate - so maybe
- we come up with a starting basic template addition on this issue
- start asking outgoing EOCs to add summary notes at the end of their shifts
- discuss/refine the process over one or two full rotations so everyone has a chance to engage at least 1-2 times
- schedule a checkpoint to revisit keeping the handover meetings or adjusting the format/duration
@gitlab-com/gl-infra WDYT?
-
the proposal is not necessarily to get rid of the handover meeting
There are kind of multiple overlapping proposals in this thread
IMO, the most important proposal, and the one that started this thread, is to put async process in place around longer-term context handover, in which the 3 outgoing EOCs summarize this information for the 3 incoming ones.
@craig's 4 points at the end of the above message are a good implementation of this, although I would prefer it if we decouple The News from the incident report. Likewise, doing the same for the sync meetings (if we keep them) would serve us well too.
If we had separate meetings for The News (chat with managers) and the on-call incident summary, I would be less likely to attend the latter because we already have a weekly sync incident review for high-impact incidents.
marked this issue as related to #12255 (moved)
mentioned in issue #12336 (closed)
marked this issue as related to #12336 (closed)
added workflow-infraDone label
moved to reliability-reports#140 (closed)
