Investigate degraded git_clone performance on 1k and 3k environments
git_clone results are degraded on 1k and 3k. The error isn't related to environment type, as the same 1k for CS has good git_clone results. The issue is to investigate what is causing slower responses on these specific test environments.
- 3k https://gitlab.com/gitlab-org/quality/performance/-/pipelines/1315654791
- 1k https://gitlab.com/gitlab-org/quality/performance/-/pipelines/1315304672
* Environment: 1k
* Environment Version: 17.1.0-pre `0b3405f5c8d`
* Option: 60s_20rps
* Date: 2024-06-03
* Run Time: 1h 27m 37.15s (Start: 04:39:31 UTC, End: 06:07:08 UTC)
* GPT Version: v2.15.0
❯ Overall Results Score: 97.24%
NAME | RPS | RPS RESULT | TTFB AVG | TTFB P90 | REQ STATUS | RESULT
---------------------------------------------------------|------|--------------------|------------|-----------------------|----------------|--------- | 20/s | 19.91/s (>16.00/s) | 101.69ms | 110.25ms (<200ms) | 100.00% (>99%) | Passed
git_clone | 1/s | 0.09/s (>0.03/s) | 995.46ms | 2507.31ms (<800ms) | 100.00% (>99%) | FAILED¹²
* Environment: 3k
* Environment Version: 17.1.0-pre `0b3405f5c8d`
* Option: 60s_60rps
* Date: 2024-06-03
* Run Time: 1h 37m 10.85s (Start: 10:22:38 UTC, End: 11:59:48 UTC)
* GPT Version: v2.15.0
❯ Overall Results Score: 98.0%
NAME | RPS | RPS RESULT | TTFB AVG | TTFB P90 | REQ STATUS | RESULT
---------------------------------------------------------|------|--------------------|-----------|-----------------------|----------------|---------
git_clone | 3/s | 0.21/s (>0.08/s) | 1425.20ms | 4441.15ms (<800ms) | 100.00% (>99%) | FAILED¹²
- https://3k.testbed.gitlab.net/-/grafana/d/J0ysCtCWz/gpt-test-results?orgId=1&var-test_name=git_clone&from=now-90d&to=now
- https://1k.testbed.gitlab.net/-/grafana/d/J0ysCtCWz/gpt-test-results?orgId=1&var-test_name=git_clone&from=now-90d&to=now
Based on historical results, the degradation started between 2024-04-22/2024-04-26 and 2024-04-29 (1k example).
- Diff:
17.0.0-pre - af5eb1269a2to17.0.0-pre - 914cf594c9b- gitlab-org/gitlab@af5eb126...914cf594