Projects::BlobController#show degrades with increased RPS
Summary
Projects::BlobController#show
(GET /:group/:project/blob/master/:file_path
) response time degrades with increased RPS if all requests are being sent to the one project on the one Gitaly node(2519.27ms). However, if requests are separated into two projects on different Gitaly nodes, the issue does not appear anymore(410.11ms ). If this is correct behavior, please clarify it in the comments below and close the issue.
Please check the results below and this comment for more context:
Results against 50k environment(1000 RPS, duration 60 sec) - 1 Gitaly node:
Environment: 50k (12.3.5-ee 9dbaa740018)
Scenario: 60s_1000rps
Date: 2019-10-10
Run Time: 2303.62s (Start: 13:40:27 UTC, End: 14:18:50 UTC)
NAME | DURATION | P95 | RPS | SUCCESS RATE | RESULT
---------------------------------------------------------|----------|------------|----------------------|----------------|-------
projects_blob_controller_show_html | 60.0s | 2519.27ms | 549.77/s (>800.00/s) | 100.00% (>95%) | Failed
Results against 50k environment(1000 RPS, duration 60 sec) - 2 Gitaly nodes:
Environment: 50k (12.3.5-ee 9dbaa740018)
Scenario: 60s_1000rps
Date: 2019-10-10
Run Time: 2303.62s (Start: 13:40:27 UTC, End: 14:18:50 UTC)
NAME | DURATION | P95 | RPS | SUCCESS RATE | RESULT
---------------------------------------------------------|----------|------------|----------------------|----------------|-------
projects_blob_controller_show_html | 60.0s | 410.11ms | 927.02/s (>800.00/s) | 100.00% (>95%) | Passed
Grafana: http://35.196.33.84/-/grafana/d/tVdO8YIWz/server-performance?orgId=1&refresh=10s
Compare with results against 10k environment(200 RPS, duration 60 sec):
Environment: 10k (12.3.5-ee 9dbaa740018)
Scenario: 60s_200rps
Date: 2019-10-11
Run Time: 1692.72s (Start: 01:42:19 UTC, End: 02:10:32 UTC)
NAME | DURATION | P95 | RPS | SUCCESS RATE | RESULT
---------------------------------------------------------|----------|------------|----------------------|----------------|-------
projects_blob_controller_show_html | 60.0s | 422.53ms | 185.98/s (>160.00/s) | 100.00% (>95%) | Passed
Grafana dashboard results: http://34.73.165.75/-/grafana/d/tVdO8YIWz/server-performance?orgId=1 (To be able to see it, please sign up)
Steps to reproduce
- Check out the Performance Toolkit
- Run the specific test with the
run-k6
command. For example against the 50k environment you would run this following from the project root:ACCESS_TOKEN="" k6/run-k6 -e environments/50k.json -s scenarios/60s_1000rps.json -t tests/projects_blob_controller_show_html.js
, whereACCESS_TOKEN
is a valid GitLab Personal Access Token for the specified environment(50k in this case). The token should come from a User that has admin access for the project(s) to be tested and have API and read_repository permissions. - If you're seeking to run the test against your own environment the Toolkit's documentation has details on how to achieve this.
What is the current bug behavior?
The API response time increases under high load on one Gitaly load.
What is the expected correct behavior?
Investigate if API response time could be improved.