Lower hurdles for notification management for EOCs

As an engineer on call (EOC) for GitLab.com, changing notification thresholds and managing our alerting rules should be easy to understand and do with confidence. Being able to do this allows an EOC to better tune noisy notifications to better optimize our time when necessary.

We should:

• Create a runbook that outlines examples on how to update our Alertmanager configurations to tune alerting thresholds.
• Create a demo that walks through how these changes can be made.
• Create a firedrill to show off this demo or walk through the runbook, or similar situation.
• Update the EOC onboarding checklist to include this runbook.