Skip to content

Blobs search API degrades with big matches count

Blobs search API's response time appears to be slow and becomes even slower if the number of code matches is quite big(8000+). Below the results of searching blobs against 10k environment:

Results summary:

Environment:    10k (12.3.3-ee 47e969b011f)
Scenario:       60s_200rps
Date:           2019-10-05
Run Time:       63.34s (Start: 15:24:19 UTC, End: 15:25:22 UTC)

NAME                                 | DURATION | P95        | RPS                 | SUCCESS RATE   | RESULT
-------------------------------------|----------|------------|---------------------|----------------|-------
api_v4_projects_project_search_blobs | 60.0s    | 18688.91ms | 10.33/s (>160.00/s) | 100.00% (>95%) | Failed
Results of the test with search query that has 300+ matches 
 Results summary:

Environment:    10k (12.3.3-ee 47e969b011f)
Scenario:       60s_200rps
Date:           2019-10-05
Run Time:       63.25s (Start: 15:49:00 UTC, End: 15:50:03 UTC)

NAME                                 | DURATION | P95       | RPS                  | SUCCESS RATE   | RESULT
-------------------------------------|----------|-----------|----------------------|----------------|-------
api_v4_projects_project_search_blobs | 60.0s    | 1377.86ms | 153.75/s (>160.00/s) | 100.00% (>95%) | Failed

In addition, this search request results in high CPU usage spike, an example of the test runs below:

Screenshot_2019-10-05_at_19.06.39

Grafana dashboard results: http://34.73.165.75/-/grafana/d/tVdO8YIWz/server-performance?orgId=1&from=1570290762899&to=1570291662899&refresh=10s (To be able to see it, please sign up)

Other sources of truth:

Steps to reproduce

  1. Check out the Performance Toolkit
  2. Update searchQuery from “test”(300+ matches) to “update”(8,500+ matches) in k6/tests/api_v4_projects_project_search_blobs.js k6/tests/api/api_v4_projects_project_search_blobs.js
  3. Run the specific test with the run-k6 command. For example against the 10k environment you would run this following from the project root: ACCESS_TOKEN="" ./run-k6 -e environments/10k.json -s scenarios/60s_200rps.json -t tests/api/api_v4_projects_project_search_blobs.js, where ACCESS_TOKEN is a valid GitLab Personal Access Token for the specified environment(10k in this case). The token should come from a User that has admin access for the project(s) to be tested and have API and read_repository permissions.
  4. If you're seeking to run the test against your own environment the Toolkit's documentation has details on how to achieve this.

What is the current bug behavior?

The API will slow depending on the number of matches in the project code.

What is the expected correct behavior?

The API responds quickly since the API only returns 20 results via pagination.

Edited by Dmitry Gruzd