Delete failed reindexing indexes created over 30 days ago

What does this MR do and why?

There are indexes for the production instance from failed reindex attempts (using zero downtime reindexing). These should have been cleaned up by the ElasticClusterReindexingCronWorker. When I looked into it, I realized the query for failed reindex tasks was incorrect.

This MR changes the old_indices_scheduled_for_deletion scope to look for:

• success reindex where the index is scheduled for deletion
• failed reindex where created_date > 30 days

It handles failed deletes different than successful deletes.

• success should delete the index_name_from (which is the old index no longer used)
• failed should delete the index_name_to (which was the new created index that was not switched over to)

The Admin UI - Search will now display failed indexes scheduled for deletion with the deletion date calculated from the created_date + 30 days (stored in a constant)

AI Summary

This code change improves how the system handles cleanup of failed search index rebuilding tasks. Previously, the system only cleaned up successful index rebuilds after a scheduled time. Now it also automatically cleans up failed index rebuilds after 30 days, preventing them from accumulating and taking up storage space. The change adds a new status to track when failed indexes are deleted, separates the cleanup logic for successful vs failed tasks, and updates the user interface to show appropriate deletion schedules for both types. It also adds safeguards to prevent duplicate cleanup jobs from running simultaneously.

References

Screenshots or screen recordings

Failed indexes scheduled for deletion

Failed indexes have been deleted

Before	After

How to set up and validate locally

1. enable advanced search/elasticsearch in gdk
2. index the instance in the admin UI or by running gitlab:elastic:index rake task
3. start a zero downtime reindexing from the Admin UI - Search - Advanced search zero-downtime reindexing
4. open a rails console to speed things up by running the cron worker manually a few times: ElasticClusterReindexingCronWorker.new.perform
5. refresh the admin UI page

I tested this with multiple data states for reindexing tasks:

• a successful and complete one
• a failed task by creating a reindexing task and letting it get to "in progress" and manually updating the task to failure state in rails console. you can set the created_at date to make sure it shows up to be deleted in the admin UI: Search::Elastic::ReindexingTask.last.update!(state: :failure, created_at: 50.days.ago)
• a failed task by manually updating only the state but do not updated created_at <-- verify it doesn't get deleted by the cron worker

Running the cronworker and making sure it cleans up the failed indexes

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.