Determine cause for gradual increase in Elasticsearch latency since 2020-11-02
Noted originally in my tracking of gitlab-com/www-gitlab-com#9559 (closed) we can see the median group scoped search latency has increased from 2s to 8s in the last month. We can't think of any specific application changes that would cause this nor do we have evidence this is happening in our performance testing against reference architectures
Data
We should have more data in the Rails Controller dashboard to help point out the problem but for some reason the Elasticsearch graph won't show anything beyond the last 7 days.
But we do have the Elastic Monitoring Cluster which seems to indicate a gradual growth in overall search latency in the last 30 days:
There doesn't seem to be any correlation in search volume growth in the last 30 days:
which implies the problem is not simply a matter of growing usage but instead the searches are just getting slower for some reason.