Automate Advanced search validation

Problem to solve

Today, search and index validations are needed when performing index and search cluster maintenance tasks. These operations are done manually.

Proposal

Add to the existing Search::ClusterHealthCheck::Elastic class to add functionality to ensure search operations are occurring without error. The new functionality must allow this check on a specific index (and should accept index name or alias).

We need to find a way to use the new health checks. Some ideas:

• Add to the migration framework to ensure search operations were not impacted by completed migrations.
  • Add search_health_status to check_and_save_completed_status method in the Elastic::MigrationWorker worker to save the search status for all indices as a first iteration.
  • Instead of saving to the migration document, write a log out after each migration is complete
• Expose prometheus metrics that run every X amount of time to validate search status per index.