GTP 1k reference architecture fails 3 tests because of failed requests
I'm in the process of migrating our self managed Gitlab instance to a new server and doing my due diligence.
We are moving to an AWS 1K reference architecture. My server is a c5.2xlarge
instance running Ubuntu 22.04.
I have installed Gitlab on the server and started it. I'm using the root user to run the tests and 3 are failing but not on the response time. The failures are with:
- api_v4_projects_issues_search
- api_v4_projects_merge_requests_search
- web_user
The GPT is running on a machine on the same VPC for which I have modified the hosts
file to use the internal VPC IP:
sudo docker run -it -e ACCESS_TOKEN=glpat-redacted -v $(pwd)/k6/config:/config -v $(pwd)/k6/tests:/tests -v $(pwd)/results:/results -v /etc/hosts:/etc/hosts gitlab/gitlab-performance-tool --environment 1k.json --options 60s_20rps.json
The following errors can be seen 5 times in the logs for api_v4_projects_issues_search
(and for api_v4_projects_merge_requests_search
):
time="2024-01-23T22:31:05Z" level=warning msg="Error detected: '{\"error\":\"This endpoint has been requested too many times. Try again later.\"}' ====> Correlation ID: 01HMW75MY06PQ77R99D58ECAWN" source=console
Which does not explain the results with 1171 failures:
* snip *
running (1m00.0s), 01/20 VUs, 1200 complete and 0 interrupted iterations
default [ 100% ] 01/20 VUs 1m00.0s/1m00.0s
█ setup
█ API - Project Issues Search
data_received..................: 3.2 MB 54 kB/s
data_sent......................: 88 kB 1.5 kB/s
group_duration.................: avg=922.09ms min=24.56ms med=999.55ms max=1298.65ms p(90)=1002.53ms p(95)=1010.31ms
http_req_blocked...............: avg=0.09ms min=0.00ms med=0.00ms max=8.01ms p(90)=0.00ms p(95)=0.00ms
http_req_connecting............: avg=0.02ms min=0.00ms med=0.00ms max=1.80ms p(90)=0.00ms p(95)=0.00ms
http_req_duration..............: avg=21.46ms min=14.03ms med=17.06ms max=314.27ms p(90)=19.45ms p(95)=45.84ms
{ expected_response:true }...: avg=128.50ms min=70.58ms med=134.79ms max=181.06ms p(90)=168.83ms p(95)=177.67ms
http_req_failed................: 97.50% ✓ 1171 ✗ 30
http_req_receiving.............: avg=0.17ms min=0.06ms med=0.07ms max=4.85ms p(90)=0.13ms p(95)=0.21ms
http_req_sending...............: avg=0.18ms min=0.06ms med=0.14ms max=2.33ms p(90)=0.26ms p(95)=0.35ms
http_req_tls_handshaking.......: avg=0.06ms min=0.00ms med=0.00ms max=6.03ms p(90)=0.00ms p(95)=0.00ms
✓ http_req_waiting...............: avg=21.11ms min=13.73ms med=16.81ms max=314.07ms p(90)=19.12ms p(95)=45.61ms
✓ http_reqs......................: 1201 20.009913/s
iteration_duration.............: avg=921.38ms min=0.15ms med=999.61ms max=1298.69ms p(90)=1002.57ms p(95)=1010.34ms
iterations.....................: 1201 20.009913/s
✗ successful_requests............: 2.49% ✓ 30 ✗ 1171
vus............................: 1 min=1 max=20
vus_max........................: 20 min=20 max=20
Here are the results, where all the Passed
results are removed:
- Environment: 1k
- Environment Version: 16.7.2-ee
847f5d82ad6
- Option: 60s_20rps
- Date: 2024-01-23
- Run Time: 1h 11m 41s (Start: 22:19:42 UTC, End: 23:31:23 UTC)
- GPT Version: v2.13.0
❯ Overall Results Score: 97.53%
NAME | RPS | RPS RESULT | TTFB AVG | TTFB P90 | REQ STATUS | RESULT |
---|---|---|---|---|---|---|
api_v4_projects_issues_search | 20/s | 20.01/s (>16.00/s) | 21.11ms | 19.12ms (<800ms) | 2.49% (>99%) | FAILED¹² |
api_v4_projects_merge_requests_search | 20/s | 20.01/s (>4.80/s) | 34.15ms | 34.43ms (<5000ms) | 2.49% (>99%) | FAILED¹² |
web_user | 2/s | 2.02/s (>0.96/s) | 32.83ms | 41.75ms (<4000ms) | 0.00% (>99%) | FAILED² |
¹ Result covers endpoint(s) that have known issue(s). Threshold(s) have been adjusted to compensate.
² Failure may not be clear from summary alone. Refer to the individual test's full output for further debugging.
Considering the RPS and TTFP results look good, should this concern me?
Is there a setup step I missed?
Thank you