Push Elasticsearch ILM policies and index templates on a schedule
Summary
This is the second time that we've found ILM policies to have been overwritten by something/someone and do not reflect what we have on git. This led to an incident as shards were not migrated properly from hot nodes to warm ones.
Until we can find out who/what is doing that, given the importance of ILM policies and index templates to be rightly configured, I'm proposing a boring workaround to push ILM policies and index templates on a schedule.
Related Incident(s)
Originating issue(s): production#7669 (closed)
Desired Outcome/Acceptance Criteria
Associated Services
Corrective Action Issue Checklist
-
Link the incident(s) this corrective action arose out of -
Give context for what problem this corrective action is trying to prevent from re-occurring -
Assign a severity label (this is the highest sev of related incidents, defaults to 'severity::4') -
Assign a priority (this will default to 'Reliability::P4')
Edited by Gonzalo Servat