Move gitlab:elastic:index to async worker

What does this MR do and why?

Related to #421298 (closed)

• switch gitlab:elastic:index rake task to run to an async Search::Elastic::TriggerIndexingWorker
• changes to Search::Elastic::TriggerIndexingWorker
  • use the rake task executor service. this will allow us to expose the same behavior from the admin UI in a future iteration
  • ensure indexing is enabled (or indexing won't work)
  • ensure indexing is paused (or index recreation may get corrupted by other activity in the system, this is the bug being fixed)
  • switch to using ApplicationSettings::UpdateService for updating application settings

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

rake tasks, will post output in testing below

How to set up and validate locally

Before

➜ echo "Feature.disable(:elastic_index_use_trigger_indexing)" | gdk rails c
➜ bundle exec rake gitlab:elastic:index
Index/alias 'gitlab-development' has been deleted
Index 'gitlab-development-issues-20240605-1424-reindex-14-0' with alias 'gitlab-development-issues' has been deleted
Index 'gitlab-development-notes-20240605-1424-reindex-14-0' with alias 'gitlab-development-notes' has been deleted
Index 'gitlab-development-merge_requests-20240605-1424-reindex-14-0' with alias 'gitlab-development-merge_requests' has been deleted
Index 'gitlab-development-commits-20240605-1424-reindex-14-0' with alias 'gitlab-development-commits' has been deleted
Index 'gitlab-development-epics-20240605-1424-reindex-14-0' with alias 'gitlab-development-epics' has been deleted
Index 'gitlab-development-users-20240605-1424-reindex-14-0' with alias 'gitlab-development-users' has been deleted
Index 'gitlab-development-wikis-20240605-1424-reindex-14-0' with alias 'gitlab-development-wikis' has been deleted
Index 'gitlab-development-projects-20240605-1424-reindex-14-0' with alias 'gitlab-development-projects' has been deleted
Index 'gitlab-development-work_items-20240605-1211' with alias 'gitlab-development-work_items' has been deleted
Index/alias 'gitlab-development-migrations' has been deleted
Index 'gitlab-development-issues-20240606-1950' has been created.
Alias 'gitlab-development-issues' -> 'gitlab-development-issues-20240606-1950' has been created.
Index 'gitlab-development-notes-20240606-1950' has been created.
Alias 'gitlab-development-notes' -> 'gitlab-development-notes-20240606-1950' has been created.
Index 'gitlab-development-merge_requests-20240606-1950' has been created.
Alias 'gitlab-development-merge_requests' -> 'gitlab-development-merge_requests-20240606-1950' has been created.
Index 'gitlab-development-commits-20240606-1950' has been created.
Alias 'gitlab-development-commits' -> 'gitlab-development-commits-20240606-1950' has been created.
Index 'gitlab-development-epics-20240606-1950' has been created.
Alias 'gitlab-development-epics' -> 'gitlab-development-epics-20240606-1950' has been created.
Index 'gitlab-development-users-20240606-1950' has been created.
Alias 'gitlab-development-users' -> 'gitlab-development-users-20240606-1950' has been created.
Index 'gitlab-development-wikis-20240606-1950' has been created.
Alias 'gitlab-development-wikis' -> 'gitlab-development-wikis-20240606-1950' has been created.
Index 'gitlab-development-projects-20240606-1950' has been created.
Alias 'gitlab-development-projects' -> 'gitlab-development-projects-20240606-1950' has been created.
Index 'gitlab-development-work_items-20240606-1950' has been created.
Alias 'gitlab-development-work_items' -> 'gitlab-development-work_items-20240606-1950' has been created.
Index '{"gitlab-development-20240606-1950"=>"gitlab-development"}' has been created.
Alias 'gitlab-development' → '{"gitlab-development-20240606-1950"=>"gitlab-development"}' has been created
Index status has been reset
Enqueuing Group level entities…
Indexing epics...
Indexing epics... done
Indexing group wikis...
Indexing group wikis... done

After

➜ echo "Feature.enable(:elastic_index_use_trigger_indexing)" | gdk rails c
➜ bundle exec rake gitlab:elastic:index
Scheduling indexing with TriggerIndexingWorker
Scheduling indexing with TriggerIndexingWorker... done

logs can be seen in logs/elasticsearch.log