Create Additional Alerts for Elasticsearch Disk Usage and Unassigned Shards
Goal
Following our recent incident, we should enhance the alerting we have in place for Elasticsearch. Specifically, we should add lower priority alerts for disk usage as well as a higher priority alert for unassigned shards.
What needs to be done
Add an additional alert condition for 70% disk usage as well as a new alert condition and panel for tracking unassigned shards to this dashboard.
QA
This should only entail adding alerts and panels to Grafana, and therefore should not require any code changes or QA.
Acceptance Criteria
-
Lower tier alert is defined for 70% disk usage for Elasticsearch data nodes -
High priority alert is defined for unassigned shards. (Note the currently 2 unassigned shards are expected)
Definition of Ready Checklist
-
Definition Of Done (DoD) -
Acceptance criteria -
Weighted -
QA