Retrying ES requests with multiple indices
Write requests in the
forward_to_all_write_targets method in
ee/lib/elastic/multi_version_util.rb currently only return the first response with errors, causing the retry logic in
ee/app/services/elastic/index_record_service.rb to retry the request on all ES indices, rather than just the failed ones.
The failures could also be different on each ES index, causing an unnoticed indexing gap.
Handle retrying per-index, so failures on each index will be checked separately and retried as necessary.
This probably involves moving the retrying logic from our Rails services into the multi-version / Gem extension layer. Maybe this could also be implemented in the upstream gems, though we're currently stuck on old versions.
Also, we currently only implement retrying for
update_document calls, but not for deletions with
delete_by_query. Retrying deletions could be split out into a separate issue.