Commit b81a25c9 authored by Shreya Shah's avatar Shreya Shah 💬
Browse files

Add documentation for Tier-1 oncall for Service teams

parent a7d7f537
Loading
Loading
Loading
Loading
+1 −0
Changes for content/handbook/engineering/infrastructure-platforms/incident-management/_index.md: 1 added line, 0 removed lines.
Original line number Diff line number Diff line
@@ -123,6 +123,7 @@ On-Call rotations notified by automated systems:
| ---- | ---- | ----------- | ---- | ---- |
| **Engineer On Call (EOC)** | [Incident Responder](./roles/incident-responder.html)| Primarily serves as the initial Incident Responder to automated alerting, and GitLab.com escalations - expectations for the role are in the [Handbook for oncall](/handbook/engineering/infrastructure-platforms/incident-management/on-call/#general-expectations-for-on-call). The checklist for the EOC is in our [runbooks](https://gitlab.com/gitlab-com/runbooks/blob/master/on-call/checklists/eoc.md). There are runbooks designed to help EOC troubleshoot a broad range of issues - in the case where the runbooks are insufficient, the EOC will escalate by [engaging the Incident Manager and CMOC](#how-to-engage-response-teams). | GitLab.com | Generally an SRE and can declare an incident. Part of the "GitLab.com Production EOC" on call schedule in incident.io. |
| **Incident Manager On Call (IMOC)** |[Incident Lead](./roles/incident-lead.html) | Provides tactical coordination and leadership during complex incidents | GitLab.com | Rotation in [incident.io](https://app.incident.io/gitlab/on-call/schedules/01K77XZFD7X7E3W8T6GDVMKAFF) |
| **Service Team Engineer On Call (STEOC)** | [Incident Responder](./roles/incident-responder.html) | Responds to automated alerting directed to their Service team for their service | GitLab.com | Rotation in incident.io |

In low severity incidents, paged individuals may play multiple roles. For example, in an S4 incident the EOC may both perform the duties of the Incident Lead and Incident Responder. As severity increases, it becomes more important to have single individuals playing these roles; individuals in [Tier 2](#tier-2) will need to be paged.

+25 −6
Changes for content/handbook/engineering/infrastructure-platforms/incident-management/on-call/tier-1.md: 25 added lines, 6 removed lines.
Original line number Diff line number Diff line
@@ -41,18 +41,37 @@ In addition to incident management responsibilities, the EOC also is responsible
- Coverage: 24x7
- Schedule: [schedule](https://gitlab.pagerduty.com/schedules#PRP4EC1)

### Runners Platform

- Rotation Leader: Kam Kyrala
- Coverage: 24x5
- Schedule: [Runners Platform on-call schedule](https://app.incident.io/gitlab/on-call/schedules/01K7HNBCW9EN2MMS4SHAJ5B2WF)

### Incident Managers (aka IMOC)

- Rotation Leader: Devin Sylva
- Coverage: 24x7
- Schedule: [schedule](https://app.incident.io/gitlab/on-call/schedules/01K77XZFD7X7E3W8T6GDVMKAFF)

## Tier-1 for Service teams (STEOC)

Service teams are paged for alerts for their owned service reducing MTTR and ensuring code ownership.

The teams are paged for alerts explicitly configured to page and carry the `Service` label that the team owns. EOC is part of the escalation chain as the first fallback for missed pages.
 
### Responsibilities 

Service teams on Tier-1 rotations are responsible for:

1. Responding to pages for alerts configured to page their owned service
2. Investigating and resolving incidents related to their owned services
3. Escalating to EOC or other Tier 2 and Tier 1 teams when the issue is outside the scope of their service or requires additional support
4. Updating alert configurations to reduce noise and improve signal quality
5. Documenting findings and resolutions in the post incident review
6. Participating in [post-incident reviews](/handbook/engineering/infrastructure-platforms/incident-review/) & creating corrective actions to improve processes and prevent recurrence

Service teams on an active Tier-1 rotation: 

#### Runners Platform

- Rotation Leader: Kam Kyrala
- Coverage: 24x5
- Schedule: [Runners Platform on-call schedule](https://app.incident.io/gitlab/on-call/schedules/01K7HNBCW9EN2MMS4SHAJ5B2WF)

#### Further details

- Incident manager rotation is staffed by certain [team members in the Engineering Group](/handbook/engineering/infrastructure-platforms/incident-management/incident-manager-onboarding/).