Skip to content

Change Pagerduty on-call to team/group effort

What is the problem?

In the last month, EMEA had roughly 40 emergencies. Sometimes multiple in quick succession of one another.

Resulting in the personal on-call having to ping Support to request that someone else handles the emergency.

Why is this a problem?

Besides being very stressful for the on-call Support Engineer. It's also quite cumbersome to have the on-call engineer having to deal with someone else taking over the other emergency.

I could list of a bunch of side effects that this could have on the Support Engineer (difference per engineer), I think we can all agree that this is not healthy for the Support Engineer that is on-call.

Proposal

I like to propose a change to the on-call schedule to have a two people being on-call at the same time. This would reduce the stress that Support Engineers experience and they could collaborate in resolving emergencies.

Required Resources

We would have to increase our subscription for Pagerduty in order to enable teams. We could get away by adding an additional schedule but this could result in sometimes a messy overlap (same person on-call in both schedules)

Desired Outcome

I hope with this change to reduce the stress level of Support Engineers and increase collaboration between Support Engineers.

The side-effect would be that instead of being on-call once every 4 months (roughly), SE's would be on-call roughly every 2 months.

Where would feedback go?

Please let me know what you think of this idea.


Some potential solutions are listed below.

  • 1️⃣ Buddy System (Always minimum of 2 SEs are on-call at all times)
    1. Rotate Daily
    2. Rotate weekly.
  • 2️⃣ Primary SE and Secondary SE (Both are on-call but if only 1 emergency comes in Primary takes it)
  • 3️⃣ Round-robin allocation from a group of SEs
  • 4️⃣ Current System (No changes)
  • 5️⃣ Primary SE and a Secondary that will be pinged by the Primary if required.

Feel free to point out if I missed any solution.

Edited by Ronald van Zon