Run gpt test suite against Ruby 3.1
As we did for Ruby 3.0 Run gpt test suite against Ruby 3 (#340294 - closed) we should run a test to check for any performance issues with Ruby 3.1
Prerequisite
MR against GitLab project with Ruby 3.1 and Omnibus package built in the MR.
Steps
- Build 10k environment using image MR above and GET
- Copy 10k configuration example under a new folder, run symlink
- Update configs to use your test GCP project. For example, using sandbox project.
- Create token on GitLab.com, it will be passed to Ansible command later to download custom GitLab package from MR
- If target package is Jammy, add
machine_image = "ubuntu-2204-lts"to enviroment.tf to switch to 22.04 (GET installs 20.04 by default) - Add
gitlab_deb_download_urlandgitlab_deb_download_url_headersfor vars.yaml to switch GET using custom Omnibus package from MR - https://gitlab.com/gitlab-org/gitlab-environment-toolkit/-/blob/main/docs/environment_configure.md#gitlab-omnibus-installation-options - example how it can look like:-
gitlab_deb_download_url: "{{ lookup('env','GITLAB_UBUNTU_IMAGE') | default('https://gitlab.com/api/v4/projects/14588374/jobs/5259591132/artifacts/pkg/ubuntu-jammy/gitlab.deb', true)}}" # update to use latest image url gitlab_deb_download_url_headers: { 'PRIVATE-TOKEN': "{{ lookup('env','PRIVATE_PROD_TOKEN')}}" } # use your .com token
-
- Build env following GET docs
- Check that Ruby version is new on Rails node - should be
3.1 - Ensure required rate limits are disabled - for GPT testing, custom Post Configure task is used https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit-configs/quality/-/blob/main/custom_task_files/gitlab_tasks/post_configure.yml#L1-35 applies it automatically (see https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit-configs/quality/-/blob/main/configs/reference_architectures/10k/ansible/inventory/vars.yml#L24-40 and https://gitlab.com/gitlab-org/gitlab-environment-toolkit/-/blob/main/docs/environment_advanced.md#custom-tasks.
- Provision test VM in the same region zone as 10k env => to trigger seed and performance testing against env from it to reduce latency
- Seed with performance data - https://gitlab.com/gitlab-org/quality/performance/-/blob/main/docs/environment_prep.md
- Horizontal Data
- Vertical Large Project Data
- Run GPT from test VM - https://gitlab.com/gitlab-org/quality/performance/-/blob/main/docs/k6.md. Explore using
screento trigger commands to ensure that command won't interrupt if your internet connection is unstable.- Running against 10k environment so using
60s_200rps.json
- Running against 10k environment so using
- Analyse results, compare it to https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/10k
- Export Server performance metrics from Grafana, share link in the issue
- Destroy env and test VM
Results
- Environment: 10k
- Environment Version: 16.6.0-pre
9f1ab4651dd - Option: 60s_200rps
- Date: 2023-10-25
- Run Time: 1h 16m 50.84s (Start: 14:09:22 UTC, End: 15:26:13 UTC)
- GPT Version: v2.13.0
❯ Overall Results Score: 97.87%
| NAME | RPS | RPS RESULT | TTFB AVG | TTFB P90 | REQ STATUS | RESULT |
|---|---|---|---|---|---|---|
| api_v4_groups | 200/s | 196.59/s (>160.00/s) | 200.91ms | 228.15ms (<200ms) | 100.00% (>99%) | FAILED² |
| api_v4_groups_group | 200/s | 161.09/s (>8.00/s) | 1127.89ms | 1669.17ms (<24000ms) | 100.00% (>20%) | Passed¹ |
| api_v4_groups_group_subgroups | 200/s | 196.4/s (>160.00/s) | 218.80ms | 249.38ms (<200ms) | 100.00% (>99%) | FAILED² |
| api_v4_groups_issues | 200/s | 97.18/s (>32.00/s) | 1909.24ms | 4710.70ms (<8000ms) | 100.00% (>99%) | Passed¹ |
| api_v4_groups_merge_requests | 200/s | 194.35/s (>16.00/s) | 328.56ms | 382.65ms (<17500ms) | 100.00% (>99%) | Passed¹ |
| api_v4_groups_projects | 200/s | 159.89/s (>8.00/s) | 1134.37ms | 1625.53ms (<22000ms) | 100.00% (>30%) | Passed¹ |
| api_v4_groups_vulnerability_report_default | 200/s | 198.09/s (>160.00/s) | 47.23ms | 53.76ms (<1000ms) | 100.00% (>99%) | Passed¹ |
| api_v4_projects | 200/s | 61.6/s (>24.00/s) | 2957.55ms | 4588.93ms (<7000ms) | 100.00% (>99%) | Passed¹ |
| api_v4_projects_deploy_keys | 200/s | 199.09/s (>160.00/s) | 52.90ms | 57.65ms (<200ms) | 100.00% (>99%) | Passed |
| api_v4_projects_issues | 200/s | 195.27/s (>160.00/s) | 265.77ms | 298.59ms (<1000ms) | 100.00% (>99%) | Passed¹ |
| api_v4_projects_issues_issue | 200/s | 194.67/s (>160.00/s) | 228.71ms | 253.46ms (<2600ms) | 100.00% (>99%) | Passed¹ |
| api_v4_projects_issues_search | 200/s | 193.7/s (>160.00/s) | 399.70ms | 560.39ms (<800ms) | 100.00% (>99%) | Passed¹ |
| api_v4_projects_languages | 200/s | 199.0/s (>160.00/s) | 50.09ms | 55.00ms (<200ms) | 100.00% (>99%) | Passed |
| api_v4_projects_merge_requests | 200/s | 197.16/s (>160.00/s) | 131.99ms | 148.50ms (<400ms) | 100.00% (>99%) | Passed¹ |
| api_v4_projects_merge_requests_merge_request | 200/s | 193.82/s (>80.00/s) | 254.07ms | 279.85ms (<3500ms) | 100.00% (>99%) | Passed¹ |
| api_v4_projects_merge_requests_merge_request_commits | 200/s | 197.88/s (>160.00/s) | 112.05ms | 127.11ms (<200ms) | 100.00% (>99%) | Passed |
| api_v4_projects_merge_requests_merge_request_diffs | 200/s | 197.75/s (>160.00/s) | 95.74ms | 108.57ms (<200ms) | 99.93% (>99%) | Passed |
| api_v4_projects_merge_requests_merge_request_discussions | 200/s | 196.65/s (>160.00/s) | 205.36ms | 233.60ms (<400ms) | 100.00% (>99%) | Passed¹ |
| api_v4_projects_merge_requests_search | 200/s | 195.53/s (>48.00/s) | 339.52ms | 538.77ms (<5000ms) | 100.00% (>99%) | Passed¹ |
| api_v4_projects_project | 200/s | 196.96/s (>160.00/s) | 128.05ms | 142.67ms (<200ms) | 100.00% (>99%) | Passed |
| api_v4_projects_project_pipelines | 200/s | 198.45/s (>160.00/s) | 68.66ms | 76.67ms (<200ms) | 100.00% (>99%) | Passed |
| api_v4_projects_project_pipelines_pipeline | 200/s | 198.07/s (>160.00/s) | 82.22ms | 89.12ms (<200ms) | 100.00% (>99%) | Passed |
| api_v4_projects_project_pipelines_pipeline_jobs | 200/s | 194.12/s (>32.00/s) | 355.13ms | 387.25ms (<2000ms) | 100.00% (>99%) | Passed¹ |
| api_v4_projects_project_services | 200/s | 199.15/s (>160.00/s) | 49.40ms | 54.72ms (<200ms) | 100.00% (>99%) | Passed |
| api_v4_projects_releases | 200/s | 197.31/s (>160.00/s) | 113.31ms | 124.13ms (<200ms) | 100.00% (>99%) | Passed |
| api_v4_projects_repository_branches | 200/s | 197.88/s (>160.00/s) | 57.72ms | 62.72ms (<200ms) | 100.00% (>99%) | Passed |
| api_v4_projects_repository_branches_branch | 200/s | 198.43/s (>160.00/s) | 87.63ms | 94.09ms (<200ms) | 100.00% (>99%) | Passed |
| api_v4_projects_repository_branches_search | 200/s | 198.04/s (>160.00/s) | 56.16ms | 61.22ms (<200ms) | 100.00% (>99%) | Passed |
| api_v4_projects_repository_commits | 200/s | 198.43/s (>160.00/s) | 87.12ms | 94.21ms (<200ms) | 100.00% (>99%) | Passed |
| api_v4_projects_repository_commits_commit | 200/s | 198.24/s (>160.00/s) | 77.07ms | 84.76ms (<200ms) | 100.00% (>99%) | Passed |
| api_v4_projects_repository_commits_commit_diff | 200/s | 196.92/s (>160.00/s) | 179.15ms | 197.28ms (<200ms) | 100.00% (>99%) | Passed |
| api_v4_projects_repository_compare | 200/s | 197.06/s (>160.00/s) | 80.06ms | 85.61ms (<200ms) | 100.00% (>99%) | Passed |
| api_v4_projects_repository_files_file | 200/s | 198.5/s (>160.00/s) | 81.20ms | 86.11ms (<200ms) | 100.00% (>99%) | Passed |
| api_v4_projects_repository_files_file_blame | 200/s | 41.52/s (>1.60/s) | 4413.05ms | 6283.35ms (<35000ms) | 100.00% (>15%) | Passed¹ |
| api_v4_projects_repository_files_file_raw | 200/s | 198.34/s (>160.00/s) | 83.57ms | 90.65ms (<200ms) | 100.00% (>99%) | Passed |
| api_v4_projects_repository_tags | 200/s | 198.16/s (>160.00/s) | 92.19ms | 98.33ms (<200ms) | 100.00% (>99%) | Passed |
| api_v4_projects_repository_tree | 200/s | 198.66/s (>160.00/s) | 84.82ms | 92.27ms (<200ms) | 100.00% (>99%) | Passed |
| api_v4_projects_vulnerability_report_filters_severity | 200/s | 198.21/s (>80.00/s) | 46.01ms | 50.68ms (<1800ms) | 100.00% (>99%) | Passed¹ |
| api_v4_projects_vulnerability_report_filters_state | 200/s | 199.27/s (>80.00/s) | 46.73ms | 51.87ms (<1800ms) | 100.00% (>99%) | Passed¹ |
| api_v4_projects_vulnerability_report_frontend_query | 200/s | 199.17/s (>80.00/s) | 45.60ms | 50.71ms (<1800ms) | 100.00% (>99%) | Passed¹ |
| api_v4_user | 200/s | 198.02/s (>160.00/s) | 48.71ms | 55.16ms (<200ms) | 100.00% (>99%) | Passed |
| api_v4_users | 200/s | 198.36/s (>160.00/s) | 63.02ms | 68.26ms (<300ms) | 100.00% (>99%) | Passed¹ |
| git_clone | 8/s | 1.03/s (>0.26/s) | 205.49ms | 468.09ms (<800ms) | 100.00% (>99%) | Passed¹ |
| git_ls_remote | 20/s | 19.89/s (>16.00/s) | 73.65ms | 79.29ms (<200ms) | 100.00% (>99%) | Passed |
| git_pull | 20/s | 19.64/s (>16.00/s) | 88.51ms | 113.48ms (<400ms) | 100.00% (>99%) | Passed |
| web_group | 20/s | 19.91/s (>16.00/s) | 184.70ms | 202.51ms (<500ms) | 100.00% (>99%) | Passed¹ |
| web_group_issues | 20/s | 19.7/s (>16.00/s) | 162.23ms | 184.14ms (<500ms) | 100.00% (>99%) | Passed¹ |
| web_group_merge_requests | 20/s | 18.83/s (>16.00/s) | 369.42ms | 674.93ms (<800ms) | 100.00% (>99%) | Passed¹ |
| web_project | 20/s | 19.36/s (>16.00/s) | 331.53ms | 356.15ms (<800ms) | 100.00% (>99%) | Passed¹ |
| web_project_branches | 20/s | 18.95/s (>16.00/s) | 420.03ms | 462.75ms (<900ms) | 100.00% (>99%) | Passed¹ |
| web_project_branches_search | 20/s | 18.6/s (>16.00/s) | 780.07ms | 825.07ms (<1600ms) | 100.00% (>99%) | Passed¹ |
| web_project_commit | 20/s | 18.99/s (>6.40/s) | 615.29ms | 1782.03ms (<3500ms) | 100.00% (>99%) | Passed¹ |
| web_project_commits | 20/s | 18.93/s (>16.00/s) | 489.03ms | 517.32ms (<1000ms) | 100.00% (>99%) | Passed¹ |
| web_project_file_blame | 20/s | 8.82/s (>3.20/s) | 1983.11ms | 2250.03ms (<2400ms) | 100.00% (>99%) | Passed¹ |
| web_project_file_rendered | 20/s | 16.18/s (>12.80/s) | 694.98ms | 751.77ms (<1700ms) | 100.00% (>99%) | Passed¹ |
| web_project_file_source | 20/s | 19.01/s (>6.40/s) | 651.48ms | 936.18ms (<4000ms) | 100.00% (>99%) | Passed¹ |
| web_project_files | 20/s | 19.48/s (>16.00/s) | 163.79ms | 255.78ms (<1200ms) | 100.00% (>99%) | Passed¹ |
| web_project_issue | 20/s | 19.44/s (>16.00/s) | 298.90ms | 522.78ms (<2000ms) | 100.00% (>99%) | Passed¹ |
| web_project_issues | 20/s | 19.57/s (>16.00/s) | 231.21ms | 244.21ms (<700ms) | 100.00% (>99%) | Passed¹ |
| web_project_issues_search | 20/s | 19.65/s (>16.00/s) | 216.13ms | 231.22ms (<700ms) | 100.00% (>99%) | Passed¹ |
| web_project_merge_request | 20/s | 18.58/s (>16.00/s) | 686.27ms | 1747.25ms (<2800ms) | 100.00% (>99%) | Passed¹ |
| web_project_merge_request_changes | 20/s | 18.48/s (>16.00/s) | 517.45ms | 708.94ms (<1500ms) | 100.00% (>99%) | Passed¹ |
| web_project_merge_request_commits | 20/s | 17.61/s (>9.60/s) | 956.24ms | 988.04ms (<1950ms) | 100.00% (>99%) | Passed¹ |
| web_project_merge_requests | 20/s | 19.17/s (>16.00/s) | 356.51ms | 399.87ms (<900ms) | 100.00% (>99%) | Passed¹ |
| web_project_merge_requests_search | 20/s | 19.25/s (>16.00/s) | 372.30ms | 387.94ms (<700ms) | 100.00% (>99%) | Passed¹ |
| web_project_pipelines | 20/s | 19.54/s (>12.80/s) | 317.04ms | 451.67ms (<600ms) | 100.00% (>99%) | Passed¹ |
| web_project_pipelines_pipeline | 20/s | 19.41/s (>16.00/s) | 475.82ms | 1026.93ms (<2500ms) | 100.00% (>99%) | Passed¹ |
| web_project_repository_compare | 20/s | 5.38/s (>0.80/s) | 3386.73ms | 4393.23ms (<7500ms) | 100.00% (>99%) | Passed¹ |
| web_project_tags | 20/s | 18.6/s (>12.80/s) | 666.66ms | 745.06ms (<1500ms) | 100.00% (>99%) | Passed¹ |
| web_user | 20/s | 19.74/s (>9.60/s) | 224.78ms | 323.34ms (<4000ms) | 100.00% (>99%) | Passed |
¹ Result covers endpoint(s) that have known issue(s). Threshold(s) have been adjusted to compensate. ² Failure may not be clear from summary alone. Refer to the individual test's full output for further debugging.%
Edited by Sofia Vistas