Runners Platform Best-Effort Tier-1 On-Call Rotation Setup

Runners Platform Best-Effort On-Call Rotation Setup

Summary

Group Name: Runners Platform
Group Manager / DRI: Kam Kyrala (@kkyrala)
Slack Channel: #g_runner_saas_infrastructure
Rotation Type: Best-Effort (24x5)

Background

The Runners Platform team is establishing a best-effort on-call rotation to provide support for runner platform issues. This is a best-effort policy, which means:

No expectation to always be at your desk
Flexible work hours are maintained - your start time does not need to shift to align with on-call
Pages are sent when you're working; if you miss a page, it escalates to others on the team
Ultimate escalation to EOC who is already on-call
24x5 coverage only (Monday-Friday, no weekends)
Team members can take breaks/walks without arranging coverage

Runners Platform On-call Members

Anton Starovoytov (@astarovoytov)
Rehab Hassanein (@rehab)
Addie Yeung (@ayeung)
Zoe Braddock (@zbraddock)
Ermia Qasemi (@ermiaqasemi)
Davis Bickford (@dbickford)
Joe Shaw (@joe-shaw)
Igor Wiedler (@igorwwwwwwwwwwwwwwwwwwww)
Tomasz Maczukin (@tmaczukin)

Runners Platform On-call Schedule

All times are in UTC Coverage is Monday through Friday only (24x5)

Schedule Structure:

Option C: We will page everyone who is available in round-robin fashion until someone responds. If no one responds, we will escalate to EOC.

Proposed Regional Coverage (to be finalized based on schedule structure)

This is a best effort rotation, so there will be no formalized "schedule" for regions.

Escalation Path

flowchart TD
    incident[Incident Escalated to Runners Platform]
    
    level1[Level 1: Page Runners Platform Team<br/>Best-effort rotation]
    
    ack_level1{Acknowledged<br/>within 15min?}
    
    eoc[Escalate to EOC<br/>Standard EOC on-call handles]
    
    resolved[Incident Handled]
    
    incident --> level1
    level1 --> ack_level1
    ack_level1 -->|Yes| resolved
    ack_level1 -->|No| eoc
    eoc --> resolved
    
    classDef level1 fill:#fff3e0
    classDef decision fill:#f3e5f5
    classDef success fill:#e8f5e8
    classDef escalation fill:#ffebee
    
    class level1 level1
    class ack_level1 decision
    class resolved success
    class eoc escalation

The default escalation path can be changed. Time intervals can be adjusted, and notification options are not fixed. If unsure, the defaults should be a reasonable starting point.

DRI Checklist

Note: For this best-effort rotation, @kkyrala (rotation leader/DRI) will be available for escalation as needed, though not in a formal Level 2 escalation chain since we escalate directly to EOC

Note: In the Escalation Path on incident.io, Notify represents Paging the folks on the schedule, Notify on Slack Channel will simply notify them on Slack

Prepare team for On-call
- Inform rotation members to ignore notifications about upcoming on-call shifts, with a message like below
```
Hi, you'll be getting a notification about upcoming on-call shifts. Do not worry, you will not be paged yet. We will only activate the rotation on date X. Any shifts scheduled before that are just for us to test the setup and prepare for the go-live.

IMPORTANT: This is a BEST-EFFORT rotation. You are NOT required to be at your desk at specific times. 
Maintain your normal flexible work hours. If you miss a page while on a break or away from your desk, 
it will escalate to others on the team and ultimately to EOC. No coverage needed for breaks/walks.
```
- Instruct your team members to set their notification preferences in the incident.io ui, this represents how they wish to be informed when they are paged
- Provide guidance on notification settings that work well with flexible schedules and best-effort expectations
- While it's not mandatory it is recommended to have the incident.io app installed on the member's mobile device (though less critical for best-effort rotation)
- Share related handbook links with the rotation members
  - Incident.io onboarding runbook
  - Incident Management Handbook
  - Document best-effort rotation expectations specific to Runners Platform team
- Review the On-Call Readiness dashboard
- Instruct the team members to finish appropriate on-call training (e.g., Tier-2 levelup course or adapted version)
- Create Runners Platform-specific on-call documentation covering:
  - Scope of runner platform on-call responsibilities
  - Common issues and escalation points
  - Best-effort expectations and guidelines
Go live!
- Set go-live date (target: after holiday season, when .com EOCs transition off that rotation)
- On the due date update the On-call teams catalog with the name (runners platform) and escalation path of your team. Each row in the catalog helps populate the drop-down menu that EOCs will use to select to page the required team.
- Announce in the #eoc-general that your team is ready to be paged, give a high-level description for this group's covered areas (runner platform infrastructure, runner managers, job execution issues, etc.), clarify this is a best-effort 24x5 rotation, and reference this handbook link (or create Runners Platform-specific documentation)
- Monitor rotation for first 2 weeks and gather feedback on notification configuration and escalation timings
- Iterate on settings based on team feedback to optimize for best-effort rotation model

Congratulations you are now ready to be on-call!

Notes:

This is establishing a best-effort, 24x5 rotation - different from traditional on-call expectations
Team members maintain flexible work hours and are not required to be available at specific times
Goal is to provide runner platform expertise without creating an undue burden on the team
EOC remains the ultimate escalation point for all incidents

workflow-infraTriage

Edited Oct 22, 2025 by Kam Kyrala