"Get raw file from repository" API degrades with increased RPS

Summary

Get raw file from repository API significantly slows down with increased RPS if all requests are being sent to the one project on the one Gitaly node(1878.52ms). However, if requests are separated into two projects on different Gitaly nodes, the issue does not appear anymore(85.83ms). It appears that tests were set a bit unrealistically with only one project in one Gitaly node, so the issue is closed.

Results against 50k environment(1000 RPS, duration 60 sec) - 2 Gitaly nodes:

Environment:    50k (12.3.5-ee 9dbaa740018)
Scenario:       60s_1000rps
Date:           2019-10-10
Run Time:       2303.62s (Start: 13:40:27 UTC, End: 14:18:50 UTC)
NAME                                                     | DURATION | P95        | RPS                  | SUCCESS RATE   | RESULT
---------------------------------------------------------|----------|------------|----------------------|----------------|-------
api_v4_projects_repository_files_file_raw                | 60.0s    | 85.83ms    | 963.28/s (>800.00/s) | 100.00% (>95%) | Passed

Results against 50k environment(1000 RPS, duration 60 sec) - 1 Gitaly node:

Environment:    50k (12.3.5-ee 9dbaa740018)
Scenario:       60s_1000rps
Date:           2019-10-10
Run Time:       2303.62s (Start: 13:40:27 UTC, End: 14:18:50 UTC)
NAME                                                     | DURATION | P95        | RPS                  | SUCCESS RATE   | RESULT
---------------------------------------------------------|----------|------------|----------------------|----------------|-------
api_v4_projects_repository_files_file_raw                | 60.0s    | 1878.52ms  | 542.9/s (>800.00/s)  | 100.00% (>95%) | Failed

Grafana: http://35.196.33.84/-/grafana/d/tVdO8YIWz/server-performance?orgId=1&refresh=10s

Compare with results against 10k environment(200 RPS, duration 60 sec):

Environment:    10k (12.3.5-ee 9dbaa740018)  
Scenario:       60s_200rps  
Date:           2019-10-11  
Run Time:       1692.72s (Start: 01:42:19 UTC, End: 02:10:32 UTC)  
  
NAME                                                     | DURATION | P95        | RPS                  | SUCCESS RATE   | RESULT  
---------------------------------------------------------|----------|------------|----------------------|----------------|-------  
api_v4_projects_repository_files_file_raw                | 60.0s    | 83.81ms    | 194.33/s (>160.00/s) | 100.00% (>95%) | Passed

Grafana dashboard results: http://34.73.165.75/-/grafana/d/tVdO8YIWz/server-performance?orgId=1 (To be able to see it, please sign up)

Steps to reproduce

  1. Check out the Performance Toolkit
  2. Run the specific test with the run-k6 command. For example against the 50k environment you would run this following from the project root: ACCESS_TOKEN="" k6/run-k6 -e environments/50k.json -s scenarios/60s_1000rps.json -t tests/api_v4_projects_repository_files_file_raw.js, where ACCESS_TOKEN is a valid GitLab Personal Access Token for the specified environment(50k in this case). The token should come from a User that has admin access for the project(s) to be tested and have API and read_repository permissions.
  3. If you're seeking to run the test against your own environment the Toolkit's documentation has details on how to achieve this.

What is the current bug behavior?

The API response time increases under high load.

What is the expected correct behavior?

That the API response time is improved.

Edited by Nailia Iskhakova