Persist Alertmanager state/config
Summary
During incident production#6538 (closed), we lost Alertmanager silences, they are not currently stored in a persistent way.
Related Incident(s)
Originating issue(s): production#6538 (closed)
Desired Outcome/Acceptance Criteria
Store Alertmanager state/config in persistent storage and/or actively backup silences.
Associated Services
Corrective Action Issue Checklist
-
Link the incident(s) this corrective action arose out of -
Give context for what problem this corrective action is trying to prevent from re-occurring -
Assign a severity label (this is the highest sev of related incidents, defaults to 'severity::4') -
Assign a priority (this will default to 'priority::4')