The mark_all_as_completed! method to drop halted cache and save halted as false

What does this MR do and why?

This MR changes the mark_all_as_completed! method to overwrite the halted state (because it should be marked as complete) and also drop the cache for whether a migration has halted.

I noticed this inconsistency in a customer support ticket when debugging another issue.

References

• https://gitlab.com/gitlab-com/request-for-help/-/issues/2850#note_2520747424

Screenshots or screen recordings

Before	After

How to set up and validate locally

1. enable advanced search
2. stop rails background jobs: gdk stop rails-background-jobs
3. open a rails console, and recreate the migrations index

     helper = Gitlab::Elastic::Helper.default
     helper.delete_migrations_index
     helper.create_migrations_index

4. run the first migration: Elastic::MigrationWorker.new.perform
5. load the admin ui: http://gdk.test:3000/admin/application_settings/search#js-elasticsearch-settings

• you will see two notifications, 1 at the top for pending migrations and 1 in Advanced search for a halted migration

6. mark all migrations as complete: ::Elastic::DataMigrationService.mark_all_as_completed!
7. the pending migration message should be gone
8. the halted migration message should be gone (on master, this message would stay around forever, even if the cache expired)

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.