## Background We are setting up a Tier 2 on-call system to provide quick, easy and reliable access to subject matter experts. The detail of why we are doing this is covered in a proposal document ([here](https://docs.google.com/document/d/1KGFMyHR2pTw_mkeOGxhG1YBY-MZDb3mwEnUd4Zdwl8M/edit?tab=t.0#heading=h.wkc7zilzdm8m)). In short, we are working on this program to manage risk and strengthen our platforms. We will also reduce our mean-time-to-recovery for incidents. ## DRI @devin ### Participants * @thisisshreya * @ahanselka ## Pilot Program We are setting up a pilot progam to establish how these Tier 2 rotations will work and how we can learn from them to eventually transition out of the pilot phase. ### Pilot Guidelines 1. Rotations should target 24x5 coverage 2. All on-call shifts occur during team-members' ordinary working hours. There is no out-of-hours requirement. This simplifies the legal requirement 3. We are using incident.io to manage the shift assignments ## Exit Criteria 1. [x] Rotation leaders have edited their rotations 2. [x] Team members have [contact methods configured](https://app.incident.io/gitlab/insights/dashboards/core/on-call-readiness?schedule%5Bone_of%5D=01JXJ7MN4T14008GQKWYYNT6E8) 3. [x] Four rotations are active with 24x5 coverage 1. After four rotations are active, we don't need to track this in an epic anymore 4. [x] All Tier 2 rotations have an escalation path 5. [x] Tier 1 EOC knows who they can escalate to, how to escalate, and in which situation that is appropriate 6. [x] We have reviewed the impact and efficacy of creating the Tier 2 pilot programs <!-- STATUS NOTE START --> ## Status 2025-10-28 🚀 **Closing status update!** 🚀 📋 **Problem:** GitLab's single-tier on-call structure hadn't scaled with growth from 54 to 93 services in two years. SREs increasingly needed subject matter expertise during incidents, impacting resolution times. 🛠️ **Solution:** Implemented four Tier 2 SME on-call rotations with established escalation paths, onboarding materials, and independently-managed schedules following standardized processes that make adding future rotations straightforward. 🎯 **Impact:** Tier 2 teams handled 52 incidents requiring specialized expertise: Gitaly (34 escalations/7 months), DBO (10 escalations/2 months), AI (2 escalations/1 month), DevOps Rails (5 escalations/2 weeks). This escalation rate validates the program's value in connecting incidents with the right domain experts quickly. 🙏 Shoutouts to the rotation leaders and all engineers participating in these on-call responsibilities! Special thanks to @igorwwwwwwwwwwwwwwwwwwww @kkyrala from the former Ops team for laying the groundwork. 🌟 Ready for closure at the next Grand Review. cc: @alex @sshah @swalker @sabrams and the NIM team 🎉 _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1600#note_2848573573_ <!-- STATUS NOTE END -->