Elasticsearch results jumping to last page gives HTTP 500

Problem to solve

Accessing higher pages in search results causes Elasticsearch to fault out by returning this error:

"Result window is too large, from + size must be less than or equal to: [10000] but was [36920]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting."

Even though this can be fixed by tweaking Elasticsearch configuration, setting index.max_result_window to a higher value probably won't perform well so we should really be looking at implementing an efficient pagination strategy.

Further details

A customer is running into this problem when accessing higher pages in Elasticsearch search results.

Proposal

We should change our currently Elasticsearch results pagination to use the scroll API.

Links / references

/cc @nick.thomas

Edited by John McGuire