feat: add retention policies for OpenSearch
Description
Adds retention policies to OpenSearch and configurable PVC size.
Supporting information
Testing instructions
Steps to test the changes:
- Check that the disk size has been changed from the default 8Gi
curl -k https://opensearch.kgdocluster.opencraft.hosting/_cat/allocation?v -uadmin:$PASSWORD
shards disk.indices disk.used disk.avail disk.total disk.percent host ip node
5 968.1kb 788.4mb 13.8gb 14.6gb 5 10.244.0.139 10.244.0.139 opensearch-cluster-master-0
3 UNASSIGNED
Check that the cleanup policy is attached to the ISM.
curl -k https://opensearch.kgdocluster.opencraft.hosting/_plugins/_ism/explain/fluent-bit*?pretty -uadmin:$PASSWORD
Expected result
{
"fluent-bit-logs" : {
"index.plugins.index_state_management.policy_id" : "cleanup-policy",
"index.opendistro.index_state_management.policy_id" : "cleanup-policy",
"index" : "fluent-bit-logs",
"index_uuid" : "4rXL6JbcS9eHs2mAXn6P9A",
"policy_id" : "cleanup-policy",
"policy_seq_no" : 0,
"policy_primary_term" : 1,
"index_creation_date" : 1683541358302,
"state" : {
"name" : "stored",
"start_time" : 1685548009534
},
"action" : {
"name" : "transition",
"start_time" : 1685548286313,
"index" : -1,
"failed" : false,
"consumed_retries" : 0,
"last_retry_time" : 0
},
"step" : {
"name" : "attempt_transition_step",
"start_time" : 1685548286315,
"step_status" : "condition_not_met"
},
"retry_info" : {
"failed" : false,
"consumed_retries" : 0
},
"info" : {
"message" : "Evaluating transition conditions [index=fluent-bit-logs]"
},
"enabled" : true
},
"total_managed_indices" : 1
}
Checklist
If any of the items below is not applicable, do not remove them, but put a check in it.
-
All providers include the new feature/change -
All affected providers can provision new clusters -
Unit tests are added/updated -
Documentation is added/updated -
The TOOLS_CONTAINER_IMAGE_VERSION
in ci_vars.yml is updated -
The grove-template repository is updated
Additional context
Add any other context about the merge request here.
Edited by Keith Grootboom