Skip to content

Determine cause for gradual increase in Elasticsearch latency since 2020-11-02

Noted originally in my tracking of gitlab-com/www-gitlab-com#9559 (closed) we can see the median group scoped search latency has increased from 2s to 8s in the last month. We can't think of any specific application changes that would cause this nor do we have evidence this is happening in our performance testing against reference architectures

Data

We should have more data in the Rails Controller dashboard to help point out the problem but for some reason the Elasticsearch graph won't show anything beyond the last 7 days.

But we do have the Elastic Monitoring Cluster which seems to indicate a gradual growth in overall search latency in the last 30 days:

Screen_Shot_2020-12-09_at_5.11.22_pm

There doesn't seem to be any correlation in search volume growth in the last 30 days:

Screen_Shot_2020-12-09_at_5.12.22_pm

which implies the problem is not simply a matter of growing usage but instead the searches are just getting slower for some reason.

Elastic support case

https://support.elastic.co/customers/s/case/5004M00000ekZRXQA2/gradual-increase-in-latency-over-last-month-leading-to-4x-slower-searches

Other resources

  1. https://www.elastic.co/blog/advanced-tuning-finding-and-fixing-slow-elasticsearch-queries
  2. https://www.elastic.co/guide/en/elasticsearch/reference/7.0/tune-for-search-speed.html
Edited by John McGuire