Optimize load times of Elasticsearch global search
ZD: https://gitlab.zendesk.com/agent/tickets/114692
A customer is load testing Elasticsearch by running what looks like JMeter tests with global search w/ Elasticsearch. The tests they are doing basically do this:
- Enter a keyword (e.g.
for,break,release,class) in the top search bar - When the results return, click on the
CodeandCommitstab in the search results.
Results in GitLab 11.5.7 before Rugged patches
In their initial tests with GitLab 11.5.7 without the Rugged patches (https://gitlab.com/gitlab-org/gitlab-ce/issues/57317), they ran several experiments:
- Slot 1: Enabling Elasticsearch with 20 active users
- Slot 2: Disabling Elasticsearch
- Slot 3: Enabling Elasticsearch with 50 active users
- Slot 4: Disabling Elasticsearch
They showed me data where some of the response times took over 100 s.
Results after Rugged patches
After applying the Rugged patches, performance significantly improved and they can test with 150 concurrent users with response times dropping 60+ seconds across the board in the 50-user case:
As the graph shows above, times still peak to about 23 s. This isn't fast enough.
There is one ruby-prof profiler in the ticket. They are going to send us additional profiles of endpoints so that we can optimize load times further.
To start, I see:
- 16% time loading CI project pipeline status:
- 17% time loading counts for milestones, blobs, projects:
- 6% LDAP access checks
/cc: @mdelaossa, @vsizov, @nick.thomas, @meks, @victorwu