Improve performance of variables API under load into main tier
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
Summary
The TTFB (Time to First Byte) of the variables API degraded on December 2022:
- List project variables API
- Creates a new project variable
- List group variables
- Creates a new group variable
- Grafana: https://10k.testbed.gitlab.net/-/grafana/d/J0ysCtCWz/gpt-test-results?orgId=1&from=now-90d&to=now&var-test_name=scenario_api_new_project_variables&var-test_name=scenario_api_new_group_variables&var-test_name=scenario_api_list_project_variables&var-test_name=scenario_api_list_group_variables
- Snapshot of Grafana dashboard: https://snapshots.raintank.io/dashboard/snapshot/6bglKgPpZmE2vYBPSt7yn1QGo7FyUnd0
This happened between 2022-12-13 and 2022-12-15
Detailed results
The TTFB (Time to First Byte) of the List project variables API is slightly over our performance target under load:
* Environment: 10k
* Environment Version: 15.7.0-pre `8b69da628bf`
* Option: 60s_200rps
* Date: 2022-12-21
* Run Time: 1h 40m 44.21s (Start: 04:46:41 UTC, End: 06:27:25 UTC)
* GPT Version: v2.12.1
NAME | RPS | RPS RESULT | TTFB AVG | TTFB P90 | REQ STATUS | RESULT
---------------------------------------------------------|-------|----------------------|-----------|-----------------------|----------------|--------
scenario_api_list_project_variables | 1/s | 1.0/s (>0.64/s) | 139.80ms | 208.11ms (<200ms) | 100.00% (>99%) | FAILED²- Test results scenario_api_new_group_variables over time, test scenario code
Note that TTFB increased from 150ms at 2022-12-14 to 200ms after 2022-12-15:
- 2022-12-14 GitLab version
15.7.0-pre - c458f253b88 - 2022-12-15 GitLab version
15.7.0-pre - 52062182e8c
Create a new group variable API
Similar degradation is seen in Creates a new group variable test:
* Environment: 10k
* Environment Version: 15.9.0-pre `22363673bf7`
* Option: 60s_200rps
* Date: 2023-01-23
* Run Time: 1h 40m 50.75s (Start: 04:50:52 UTC, End: 06:31:43 UTC)
* GPT Version: v2.12.2
NAME | RPS | RPS RESULT | TTFB AVG | TTFB P90 | REQ STATUS | RESULT
---------------------------------------------------------|-------|----------------------|-----------|-----------------------|----------------|--------
scenario_api_new_group_variables | 1/s | 1.01/s (>0.64/s) | 134.43ms | 212.02ms (<200ms) | 100.00% (>99%) | FAILED²- Changes - 52062182...5585d088
Create a new project variable API
Similar degradation is seen in Creates a new project variable test - the results becam more spiky after December 2022:
* Environment: 10k
* Environment Version: 15.9.0-pre `9a7ab190daf`
* Option: 60s_200rps
* Date: 2023-01-24
* Run Time: 1h 38m 25.23s (Start: 04:47:58 UTC, End: 06:26:23 UTC)
* GPT Version: v2.12.2
❯ Overall Results Score: 97.74%
NAME | RPS | RPS RESULT | TTFB AVG | TTFB P90 | REQ STATUS | RESULT
---------------------------------------------------------|-------|----------------------|-----------|-----------------------|----------------|--------
scenario_api_new_project_variables | 1/s | 1.01/s (>0.64/s) | 144.45ms | 235.85ms (<200ms) | 100.00% (>99%) | FAILED²Test Details
Testing was done on our 10k Reference Architecture environment with our lab condition GitLab Performance Tool pipeline. The project being tested is a copy of gitlabhq (tarball can be found here). GitLab Performance Tool tests information is listed at Current test details page.
The latest GitLab Performance pipeline results can always be found here. Through this page full Server Metrics can be found via the Metrics Dashboard link on that page.
As per our performance targets this endpoint's TTFB metric is above the target of 200 ms which is severity4. Task is to improve the endpoint's performance into next tier.