Advanced Search: Move issues to new Elasticsearch index (not revertable)

What does this MR do?

NOTE: Since this MR moves issues to a new Elasticsearch index and since updates will only be going to that new index it's not easy to simply revert this MR. If we did revert this we'd not have any issues as search results. Ideally we'd roll forward. In the event that something did go wrong we could revert it but we'd need to schedule follow up work to correct all the issues in the index and issue search would likely be missing results until we fixed that.

This MR does a few things:

• adds an ability to pause indexing for the duration of a migration via pause_indexing! setting
• introduces an ability to pass a hash between different runs of a batched migration, this is called launch_options
• creates a migration to move all issues from the main index to a separate one called gitlab-production-issues using manual slices
• after the migration is completed it switches indexing and querying issues to using a newly created index

#273264 (closed)

Merge order

1. !49617 (merged) -> 273264-copy-issues-to-new-index
2. !49418 (merged) -> 273479-update-create_empty_index-rake-task
3. !49370 (merged) -> 273264-copy-issues-to-new-index
4. !50267 (merged) -> 273264-copy-issues-to-new-index
5. this MR -> master

Timing

The new migration MigrateIssuesToSeparateIndex should take around 2 hrs to complete on our index after it starts.

It has throttle_delay 1.minute and it will run one time per slice and max_slices: get_number_of_shards and our production cluster has 120 shards so therefore we'd expect it to run for around 120 minutes = 2 hrs.

Screenshots (strongly suggested)

Useful links

• https://github.com/elastic/elasticsearch-rails/tree/master/elasticsearch-model
• https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html
• https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html#slice-scroll
• gitlab-com/gl-infra/production#2530 (closed)

Does this MR meet the acceptance criteria?

Conformity

• Changelog entry
• [-] Documentation (if required)
• Code review guidelines
• Merge request performance guidelines
• Style guides
• [-] Database guides
• [-] Separation of EE specific content

Availability and Testing

• Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process.
• [-] Tested in all supported browsers
• [-] Informed Infrastructure department of a default or new setting change, if applicable per definition of done

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

• [-] Label as security and @ mention @gitlab-com/gl-security/appsec
• [-] The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
• [-] Security reports checked/validated by a reviewer from the AppSec team

Related to #273264 (closed)