Automate oncall handover issue creation
Overview
Since we switched to our 8-hour oncall rotation, we started creating OnCall Handover issues on the on-call-handover queue. So far, we have created 272 issues and all of these were done manually. If it takes about 5 min to create an issue (because we have to search for CRs, incidents, alert links...etc) then we spent about 1360 mins (22.6 hours) so far. In a 24 hour period, we open 3 issues. That projects us to create 1095 issues by the end of the year and we will have spent 5475 mins -- which is nearly 91.25 hours. So perhaps, if we can automate some of this process, it could save us time.
Current Process
Currently, what we do is:
- Go to https://gitlab.com/gitlab-com/gl-infra/on-call-handovers/issues
- Create a new issue and select the
handover
template - Fill out the template by copying / pasting issue links (CRs, incidents if any), alert llinks...etc. And provide a summary
- Co-relate the previous issue
Automation
Considerations:
- Given that the handover issue does require some manual input such as the
Summary
portion, the whole thing doesn't appear to be automatable in this iteration. - The exact time at which OnCalls create their handover issue seems to be inconsistent (i.e not everyone creates an issue at the exact 8th hour of their shift, though we try to).
Therefore, giving the issue creation control to the OnCall makes sense but creation in the sense that just click a button or run a simple command to trigger the automation work and still be able to provide some manual input.
As for the automatable tasks:
- Pull ongoing alerts, if any
- Pull ongoing incidents, if any
- Pull resolved + actionable alerts, if any
- Pull unactionable alerts
- Pull resolved production incident issues, if any
- Pull change requests, if any
and all preferably in the last 8 hour. But if oncall creates the issue prior to the end of their shift, then this last-8-hour logic would not work very well. (But we can iterate on it)
Implementation
An idea I have is:
- Create a Slack app
- Configure a workflow on it or interactive components
- Allow OnCall type a command or click a button to start the app
- The app should query issues, CRs, alerts it can, prepopulate a template
- Let OnCall add any missing info such as summary
- Save and it should create an issue in the on-call-handover queue