Skip to content

GTP 1k reference architecture fails 3 tests because of failed requests

I'm in the process of migrating our self managed Gitlab instance to a new server and doing my due diligence.

We are moving to an AWS 1K reference architecture. My server is a c5.2xlarge instance running Ubuntu 22.04.

I have installed Gitlab on the server and started it. I'm using the root user to run the tests and 3 are failing but not on the response time. The failures are with:

  • api_v4_projects_issues_search
  • api_v4_projects_merge_requests_search
  • web_user

The GPT is running on a machine on the same VPC for which I have modified the hosts file to use the internal VPC IP:

sudo docker run -it -e ACCESS_TOKEN=glpat-redacted -v $(pwd)/k6/config:/config -v  $(pwd)/k6/tests:/tests -v $(pwd)/results:/results -v /etc/hosts:/etc/hosts  gitlab/gitlab-performance-tool --environment 1k.json --options 60s_20rps.json

The following errors can be seen 5 times in the logs for api_v4_projects_issues_search (and for api_v4_projects_merge_requests_search):

time="2024-01-23T22:31:05Z" level=warning msg="Error detected: '{\"error\":\"This endpoint has been requested too many times. Try again later.\"}' ====> Correlation ID: 01HMW75MY06PQ77R99D58ECAWN" source=console

Which does not explain the results with 1171 failures:

* snip * 
running (1m00.0s), 01/20 VUs, 1200 complete and 0 interrupted iterations
default   [ 100% ] 01/20 VUs  1m00.0s/1m00.0s

     █ setup

     █ API - Project Issues Search

     data_received..................: 3.2 MB 54 kB/s
     data_sent......................: 88 kB  1.5 kB/s
     group_duration.................: avg=922.09ms min=24.56ms med=999.55ms max=1298.65ms p(90)=1002.53ms p(95)=1010.31ms
     http_req_blocked...............: avg=0.09ms   min=0.00ms  med=0.00ms   max=8.01ms    p(90)=0.00ms    p(95)=0.00ms 
     http_req_connecting............: avg=0.02ms   min=0.00ms  med=0.00ms   max=1.80ms    p(90)=0.00ms    p(95)=0.00ms 
     http_req_duration..............: avg=21.46ms  min=14.03ms med=17.06ms  max=314.27ms  p(90)=19.45ms   p(95)=45.84ms
       { expected_response:true }...: avg=128.50ms min=70.58ms med=134.79ms max=181.06ms  p(90)=168.83ms  p(95)=177.67ms
     http_req_failed................: 97.50% ✓ 1171      ✗ 30
     http_req_receiving.............: avg=0.17ms   min=0.06ms  med=0.07ms   max=4.85ms    p(90)=0.13ms    p(95)=0.21ms 
     http_req_sending...............: avg=0.18ms   min=0.06ms  med=0.14ms   max=2.33ms    p(90)=0.26ms    p(95)=0.35ms 
     http_req_tls_handshaking.......: avg=0.06ms   min=0.00ms  med=0.00ms   max=6.03ms    p(90)=0.00ms    p(95)=0.00ms 
   ✓ http_req_waiting...............: avg=21.11ms  min=13.73ms med=16.81ms  max=314.07ms  p(90)=19.12ms   p(95)=45.61ms
   ✓ http_reqs......................: 1201   20.009913/s
     iteration_duration.............: avg=921.38ms min=0.15ms  med=999.61ms max=1298.69ms p(90)=1002.57ms p(95)=1010.34ms
     iterations.....................: 1201   20.009913/s
   ✗ successful_requests............: 2.49%  ✓ 30        ✗ 1171
     vus............................: 1      min=1       max=20
     vus_max........................: 20     min=20      max=20

Here are the results, where all the Passed results are removed:

  • Environment: 1k
  • Environment Version: 16.7.2-ee 847f5d82ad6
  • Option: 60s_20rps
  • Date: 2024-01-23
  • Run Time: 1h 11m 41s (Start: 22:19:42 UTC, End: 23:31:23 UTC)
  • GPT Version: v2.13.0

❯ Overall Results Score: 97.53%

NAME RPS RPS RESULT TTFB AVG TTFB P90 REQ STATUS RESULT
api_v4_projects_issues_search 20/s 20.01/s (>16.00/s) 21.11ms 19.12ms (<800ms) 2.49% (>99%) FAILED¹²
api_v4_projects_merge_requests_search 20/s 20.01/s (>4.80/s) 34.15ms 34.43ms (<5000ms) 2.49% (>99%) FAILED¹²
web_user 2/s 2.02/s (>0.96/s) 32.83ms 41.75ms (<4000ms) 0.00% (>99%) FAILED²

¹ Result covers endpoint(s) that have known issue(s). Threshold(s) have been adjusted to compensate.
² Failure may not be clear from summary alone. Refer to the individual test's full output for further debugging.

Considering the RPS and TTFP results look good, should this concern me?
Is there a setup step I missed?

Thank you