Use elastic retry setting for search requests

What does this MR do and why?

Follow-up from Add application setting for elasticsearch retry... (!169903 - merged) which introduced an application setting called elasticsearch_retry_on_failure. This MR uses that setting to retry search requests when they fail. This only applies to search requests, not index, maintanence, deletes, etc.

Screenshots or screen recordings

Searches that go via the proxies:

Searches that use the execute_search method:

References

Please include cross links to any resources that are relevant to this MR This will give reviewers and future readers helpful context to give an efficient review of the changes introduced.

• !169903 (merged)
• #486935 (closed)
• https://www.elastic.co/guide/en/elasticsearch/client/ruby-api/current/advanced-config.html

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

How to set up and validate locally

1. Run migrations
2. Run a rails console with ELASTIC_CLIENT_DEBUG=true
3. Update the setting to be non-zero: settings = ApplicationSetting.current; settings.elasticsearch_retry_on_failure = 5; settings.save!
4. Set the elasticsearch url to something that doesn't exist so that requests fail: settings = ApplicationSetting.current; settings.elasticsearch_url = "http://localhost:9300"; settings.save!
5. Run a search that uses a proxy: ::SearchService.new(User.first, {scope: 'projects', search: '*'}).search_objects
   1. See that the request fails and retries 5 times
6. Run a search that uses the execute_search method: ::SearchService.new(User.first, {scope: 'issues', search: '*'}).search_objects
   1. See that the request fails and retries 5 times
7. Do the same for a non-search query, e.g. Gitlab::Elastic::Helper.default.client.ping
   1. See that the request fails and does not retry
8. Update the setting to be 0: settings = ApplicationSetting.current; settings.elasticsearch_retry_on_failure = 0; settings.save!
9. Run a search that uses a proxy: ::SearchService.new(User.first, {scope: 'projects', search: '*'}).search_objects
   1. See that the request fails and does not retry
10. Run a search that uses the execute_search method: ::SearchService.new(User.first, {scope: 'issues', search: '*'}).search_objects
    1. See that the request fails and does not retry
11. Test with a correct elasticsearch URL and see that everything works as normal

Related to #486935 (closed)