Add a way to temporarily store logs for projects deleted out of the Elasticsearch index for several months.

Note: This issue replaced Add audit event for ElasticDeleteProjectWorker

Overview

In order to better track and analyze problems with the Elasticsearch projects that are deleted we need to store logs for the ElasticDeleteProjectWorker for several months.

Problem

In order to better debug issues on index integrity, it would be helpful to have a longer period of log retention to diagnose the problem. Unfortunately, the current retention is 7 days and support cases don't reach the groupglobal search team until 10-20 days later than the incident.

Proposal

Create a watcher event to move ElasticDeleteProjectWorker to a separate index with longer ILM retention

In order to minimize costs, this will be added to a separate index with warm hardware and a retention period of 90 days, we will be able to diagnose issues more efficiently.

Eventually, we might not need this index and can delete it.

Risks

We assume that this new logging index will not be very large and would mostly fit inside the existing storage allocated for logging. We think the risk is low for anything related to impacting logging performance or storage needs related to the logging cluster but can periodically review if this causes an issue.