Skip to content

feat: add retention policies for OpenSearch

Keith Grootboom requested to merge keith/opensearch-ism into main

Description

Adds retention policies to OpenSearch and configurable PVC size.

Supporting information

Testing instructions

Steps to test the changes:

  1. Check that the disk size has been changed from the default 8Gi
curl -k https://opensearch.kgdocluster.opencraft.hosting/_cat/allocation?v -uadmin:$PASSWORD
shards disk.indices disk.used disk.avail disk.total disk.percent host         ip           node
     5      968.1kb   788.4mb     13.8gb     14.6gb            5 10.244.0.139 10.244.0.139 opensearch-cluster-master-0
     3                                                                                     UNASSIGNED

Check that the cleanup policy is attached to the ISM.

curl -k https://opensearch.kgdocluster.opencraft.hosting/_plugins/_ism/explain/fluent-bit*?pretty -uadmin:$PASSWORD
Expected result
{
  "fluent-bit-logs" : {
    "index.plugins.index_state_management.policy_id" : "cleanup-policy",
    "index.opendistro.index_state_management.policy_id" : "cleanup-policy",
    "index" : "fluent-bit-logs",
    "index_uuid" : "4rXL6JbcS9eHs2mAXn6P9A",
    "policy_id" : "cleanup-policy",
    "policy_seq_no" : 0,
    "policy_primary_term" : 1,
    "index_creation_date" : 1683541358302,
    "state" : {
      "name" : "stored",
      "start_time" : 1685548009534
    },
    "action" : {
      "name" : "transition",
      "start_time" : 1685548286313,
      "index" : -1,
      "failed" : false,
      "consumed_retries" : 0,
      "last_retry_time" : 0
    },
    "step" : {
      "name" : "attempt_transition_step",
      "start_time" : 1685548286315,
      "step_status" : "condition_not_met"
    },
    "retry_info" : {
      "failed" : false,
      "consumed_retries" : 0
    },
    "info" : {
      "message" : "Evaluating transition conditions [index=fluent-bit-logs]"
    },
    "enabled" : true
  },
  "total_managed_indices" : 1
}

Checklist

If any of the items below is not applicable, do not remove them, but put a check in it.

  • All providers include the new feature/change
  • All affected providers can provision new clusters
  • Unit tests are added/updated
  • Documentation is added/updated
  • The TOOLS_CONTAINER_IMAGE_VERSION in ci_vars.yml is updated
  • The grove-template repository is updated

Additional context

Add any other context about the merge request here.

Edited by Keith Grootboom

Merge request reports