Retrying ES requests with multiple indices

Summary

This is a follow-up to !17230 (closed) / !18254 (merged).

Write requests in the forward_to_all_write_targets method in ee/lib/elastic/multi_version_util.rb currently only return the first response with errors, causing the retry logic in ee/app/services/elastic/index_record_service.rb to retry the request on all ES indices, rather than just the failed ones.

The failures could also be different on each ES index, causing an unnoticed indexing gap.

Improvements

Handle retrying per-index, so failures on each index will be checked separately and retried as necessary.

This probably involves moving the retrying logic from our Rails services into the multi-version / Gem extension layer. Maybe this could also be implemented in the upstream gems, though we're currently stuck on old versions.

Also, we currently only implement retrying for es_import / index_document / update_document calls, but not for deletions with delete_document, delete_child_documents, delete_by_query. Retrying deletions could be split out into a separate issue.

Involved components

• ee/lib/elastic/multi_version_util.rb
• ee/app/services/elastic/index_record_service.rb