Verification of git reftables
Support reftables in Gitaly (gitlab-org&11653 - closed) tracks the work relating to the changes needed within gitaly to support the new reftables backend.
Part of this effort will likely require making some changes to some configuration to enable this feature. We'll want to create a new job to help verify these changes in gitlab-qa, and in the future potentially make this the default behaviour.
Similar to the praefect job, we will probably only need to run E2E tests that interact with gitaly, so we can scope this effort to a smaller subset of tests.
Designs
- Show closed items
Activity
-
Newest first Oldest first
-
Show all activity Show comments only Show history only
- John McDonnell added to epic gitlab-org&11653 (closed)
added to epic gitlab-org&11653 (closed)
- Patrick Steinhardt added gitreftable label
added gitreftable label
- Karthik Nayak changed epic to gitlab-org&12503
changed epic to gitlab-org&12503
- Developer
Hey @john.mcdonnell, is there something we can do to get this going? Would be nice to have this before we start the rollout onto production.
Collapse replies - Author Maintainer
Hey @knayakgl - sure thing. I've added this into the current milestone and I'll look into explicitly what we need to make this happen.
- Author Maintainer
Hi @knayakgl - I've run an initial set of GPT tests against an environment to do a comparison of performance between
ref-format=files/reftable
As noted, there does appear to be some noticeable degradation of performance on
api_v4_projects_repository_commits_all
that might be of note. Would you have any thoughts on why we might be seeing such as result here, and if indeed it is something you can reproduce.
- John McDonnell changed milestone to %17.4
changed milestone to %17.4
- John McDonnell added gitlab-org/quality#74 as parent epic
added gitlab-org/quality#74 as parent epic
- Kassandra Svoboda assigned to @john.mcdonnell
assigned to @john.mcdonnell
- Andras Horvath added workflowin dev label
added workflowin dev label
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#19542 (closed)
mentioned in issue gitlab-org/quality/triage-reports#19542 (closed)
- Author Maintainer
From an end user perspective, this will not be overly visible so we shouldn't need too much work from a new E2E test perspsective, but it does of course have the potential to impact a wide range of features so for this effort I'd suggest we should:
- in similar fashion to Support gitaly transactions pipeline (gitlab-org/gitlab!152388 - merged) we should deploy a new E2E test pipeline with this feature flag enabled, with a subsequent issue scheduled to remove the FF pipeline once the feature has been fully rolled out
- we should add a singe new E2E test to verify that the repository being created when the feature flag is enabled use the retable backend. In the future this could be modified to verify that the file backend can also be used if that remains an option after this is enabled.
- Running a one off manual GPT suite to compare performance using reftables to the files backend. We should be able to use
git refs migrate --ref-format=reftable
command to migrate the repository under test manually. - When this feature is set to be the default option, we will also want to consider updating GPT test data to use this backend
Details of the Feature Flag itself are available in gitlab-org/gitaly@23a6e749
Edited by John McDonnell - John McDonnell mentioned in merge request gitlab-org/gitlab!165026 (merged)
mentioned in merge request gitlab-org/gitlab!165026 (merged)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#19657 (closed)
mentioned in issue gitlab-org/quality/triage-reports#19657 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#19669 (closed)
mentioned in issue gitlab-org/quality/triage-reports#19669 (closed)
- Karthik Nayak mentioned in merge request gitlab-com/content-sites/handbook!8008 (merged)
mentioned in merge request gitlab-com/content-sites/handbook!8008 (merged)
- Author Maintainer
Initial GPT Results
Environment Info
- GitLab Version:
17.4.0-pre f6354917596
- The environment was setup as a 3k reference architecture: https://docs.gitlab.com/ee/administration/reference_architectures/3k_users.html
- GPT running version 79a2ab41
Test Execution Notes
- GPT test 3x with , before manually updating the repository to use the ref table format on each of the gitaly nodes, gitaly_new_repo_reftable_backend enabled, and 3x GPT test run again. This should help us spot any data outliers.
Steps taken to update environment when toggling between the ref-format
gitaly-1 / gitaly-2 / gitaly-3 $ sudo su $ cd /var/opt/gitlab/git-data/repositories/@cluster/repositories/e4/43/10001 $ du -sh * 4.0K HEAD 4.0K config 878M objects 852K packed-refs 12K refs $ git config --global --add safe.directory /var/opt/gitlab/git-data/repositories/@cluster/repositories/e4/43/10001 $ /opt/gitlab/embedded/bin/gitaly-git-v2.46 refs migrate --ref-format=reftable $ du -sh * 4.0K HEAD 4.0K config 878M objects 8.0K refs 628K reftable gitlab-rails-1 / gitlab-rails-2 / gitlab-rails-3 $ sudo gitlab-rails console Feature.enable(:gitaly_new_repo_reftable_backend) => true irb(main):001:0> Feature.enabled?(:gitaly_new_repo_reftable_backend) => true
Results
-
The main standout item in these results is
api_v4_projects_repository_commits_all
which targets the/api/v4/projects/${project['encoded_path']}/repository/commits?all=true
endpoint.
With ref-format=files this Avg: 382 / P90: 758 / P95: 1015, but this appears to have increased by at least 2x-4x while using ref-format=reftable backend. -
We also observe a slight degradation in performance on endpoints such as:
Further analysis on
api_v4_projects_repository_commits_all
In order to verify that this isn't an anomaly, I re ran this specific test a number of times, toggling between the ref-format files/reftables, and confirmed that this appears to be a reproducible result
/opt/gitlab/embedded/bin/gitaly-git-v2.46 refs migrate --ref-format=files ❯ Overall Results Score: 51.99% NAME | RPS | RPS RESULT | TTFB AVG | TTFB P90 | REQ STATUS | RESULT ---------------------------------------|------|--------------------|-----------|---------------------|---------------|------------------ api_v4_projects_repository_commits_all | 60/s | 43.18/s (>48.00/s) | 4680.46ms | 11806.69ms (<200ms) | 72.24% (>99%) | FAILED¹² NOTE HERE: Only 72.24% of requests are successful. Upon investigation I noted that the refs are not packed so we have a directory that looks like: 4.0K HEAD 4.0K config 878M objects 37M refs This seems like it would be a reasonable explanation for the poor performance here, suggesting that this might be something that we might want to document, to ensure that anyone switching from reftable to files, should also ensure to run something like `pack-refs` at the same time.
With ref-format=files, I ran pack-refs, which gave a better result. /opt/gitlab/embedded/bin/gitaly-git-v2.46 pack-refs --all 4.0K HEAD 4.0K config 865M objects 852K packed-refs 704K refs ❯ Overall Results Score: 91.92% NAME | RPS | RPS RESULT | TTFB AVG | TTFB P90 | REQ STATUS | RESULT ---------------------------------------|------|--------------------|----------|--------------------|----------------|------------------ api_v4_projects_repository_commits_all | 60/s | 55.15/s (>48.00/s) | 439.43ms | 1057.55ms (<200ms) | 100.00% (>99%) | FAILED¹²
/opt/gitlab/embedded/bin/gitaly-git-v2.46 refs migrate --ref-format=reftable 4.0K HEAD 4.0K config 878M objects 8.0K refs 628K reftable ❯ Overall Results Score: 91.33% NAME | RPS | RPS RESULT | TTFB AVG | TTFB P90 | REQ STATUS | RESULT ---------------------------------------|------|-------------------|----------|--------------------|----------------|------------------ api_v4_projects_repository_commits_all | 60/s | 54.8/s (>48.00/s) | 701.00ms | 1669.03ms (<200ms) | 100.00% (>99%) | FAILED¹²
Edited by John McDonnell - GitLab Version:
Collapse replies - Author Maintainer
Results comparison provided as follows:
Columns
files1/file2/file3
represent the 3 runs using ref-format=files Columnsreftable1/reftable2/reftable3
represent the 3 runs using ref-format=reftableTTFB Avg (ms)
TTFB Avg (ms) Name files1 files2 file3 reftable1 reftable2 reftable3 Average Files Average Reftable Diff api_v4_groups 1159 1164 1159 1063 1108 1122 1161 1098 -5.42% api_v4_groups_group 2665 2640 2665 2384 2448 2835 2657 2555 -3.81% api_v4_groups_group_subgroups 1188 1162 1188 1110 1153 1190 1179 1151 -2.41% api_v4_groups_issues 2118 2101 2118 2105 2159 2126 2113 2130 0.82% api_v4_groups_merge_requests 371 368 371 323 334 369 370 342 -7.52% api_v4_groups_projects 2404 2367 2404 2403 2330 2771 2391 2501 4.59% api_v4_projects 5718 5775 5718 5335 5338 5601 5737 5425 -5.45% api_v4_projects_deploy_keys 56 55 56 54 54 56 55 54 -1.89% api_v4_projects_issues 253 252 253 237 238 271 253 249 -1.68% api_v4_projects_issues_issue 335 321 335 329 317 341 330 329 -0.41% api_v4_projects_issues_search 334 281 334 310 412 288 316 337 6.54% api_v4_projects_languages 54 53 54 54 54 58 54 55 3.33% api_v4_projects_merge_requests 156 157 156 158 150 151 156 153 -2.08% api_v4_projects_merge_requests_merge_request 654 590 654 778 506 618 633 634 0.14% api_v4_projects_merge_requests_merge_request_commits 117 116 117 118 111 114 117 114 -1.98% api_v4_projects_merge_requests_merge_request_diffs 99 97 99 103 97 97 98 99 0.27% api_v4_projects_merge_requests_merge_request_discussions 232 218 232 250 225 216 227 230 1.17% api_v4_projects_merge_requests_search 1124 1107 1124 1117 1107 1100 1118 1108 -0.85% api_v4_projects_project 174 167 174 169 169 171 171 170 -1.09% api_v4_projects_project_pipelines 75 72 75 71 70 72 74 71 -4.24% api_v4_projects_project_pipelines_pipeline 92 86 92 85 85 86 90 85 -5.11% api_v4_projects_project_pipelines_pipeline_jobs 1254 1196 1254 1183 1189 1226 1235 1199 -2.86% api_v4_projects_project_services 79 76 79 74 73 75 78 74 -4.77% api_v4_projects_releases 126 124 126 120 117 120 125 119 -4.85% api_v4_projects_repository_branches 79 78 79 79 76 79 79 78 -0.93% api_v4_projects_repository_branches_branch 120 119 120 126 119 120 120 122 1.85% api_v4_projects_repository_branches_search 79 77 79 78 74 77 78 76 -2.18% api_v4_projects_repository_commits 112 111 112 118 114 117 112 117 4.41% api_v4_projects_repository_commits_all 385 374 385 3306 722 1794 382 1941 408.62% api_v4_projects_repository_commits_commit 110 108 110 109 103 106 109 106 -3.22% api_v4_projects_repository_commits_commit_diff 233 233 233 278 227 235 233 247 5.93% api_v4_projects_repository_compare 113 124 113 119 109 118 117 116 -0.99% api_v4_projects_repository_files_file_blame 10123 10702 10123 11700 11328 11856 10316 11628 12.72% api_v4_projects_repository_files_file_metadata 119 121 119 120 116 125 119 120 0.58% api_v4_projects_repository_files_file_raw 127 123 127 128 122 129 126 126 0.70% api_v4_projects_repository_tags 125 123 125 124 120 128 125 124 -0.48% api_v4_projects_repository_tree 127 127 127 128 123 132 127 128 0.39% git_clone 214 227 214 217 222 211 218 216 -0.86% git_ls_remote 87 84 87 92 96 109 86 99 14.93% git_pull 123 120 123 128 130 125 122 128 4.87% web_group 213 255 213 232 220 244 227 232 2.40% web_group_issues 216 197 216 187 183 198 210 189 -9.65% web_group_merge_requests 349 327 349 332 342 354 342 343 0.29% web_project 292 305 292 288 292 304 296 295 -0.54% web_project_branches 428 457 428 433 439 469 437 447 2.23% web_project_branches_search 827 835 827 801 809 849 830 820 -1.23% web_project_commit 613 607 613 611 604 615 611 610 -0.16% web_project_commits 524 542 524 512 522 523 530 519 -2.07% web_project_file_blame 2528 2625 2528 2544 2525 2566 2560 2545 -0.59% web_project_file_rendered 516 516 516 530 502 510 516 514 -0.29% web_project_file_source 552 563 552 557 534 555 556 549 -1.25% web_project_files 171 168 171 175 171 164 170 170 0.15% web_project_issue 303 296 303 310 301 296 301 302 0.44% web_project_issues 253 249 253 266 246 256 252 256 1.77% web_project_issues_search 245 257 245 256 241 241 249 246 -1.23% web_project_merge_request 686 688 686 696 670 679 687 682 -0.78% web_project_merge_request_changes 503 514 503 511 496 505 507 504 -0.59% web_project_merge_request_commits 806 845 806 838 800 816 819 818 -0.16% web_project_merge_requests 313 342 313 329 312 329 323 323 0.22% web_project_merge_requests_search 332 356 332 342 333 332 340 335 -1.37% web_project_pipelines 350 375 350 368 358 361 358 362 1.17% web_project_pipelines_pipeline 657 666 657 656 642 635 660 644 -2.31% web_project_repository_compare 3408 3412 3408 3466 3384 3413 3409 3421 0.35% web_project_tags 720 769 720 752 728 752 736 744 1.10% TTFB P90 (ms)
TTFB P90 (ms) Name files1 files2 file3 reftable1 reftable2 reftable3 Average Files Average Reftable Diff api_v4_groups 1314 1387 1314 1185 1280 1236 1338 1233 -7.84% api_v4_groups_group 3115 3268 3115 2720 2786 3954 3166 3154 -0.39% api_v4_groups_group_subgroups 1370 1367 1370 1245 1302 1450 1369 1332 -2.69% api_v4_groups_issues 4049 3199 4049 3276 3070 3594 3766 3313 -12.02% api_v4_groups_merge_requests 481 479 481 410 426 484 480 440 -8.37% api_v4_groups_projects 2723 2720 2723 2823 2657 4685 2722 3388 24.47% api_v4_projects 6946 7655 6946 6541 6566 6901 7182 6669 -7.14% api_v4_projects_deploy_keys 59 58 59 57 57 62 59 58 -0.70% api_v4_projects_issues 328 331 328 288 295 364 329 316 -4.02% api_v4_projects_issues_issue 574 568 574 563 545 545 572 551 -3.68% api_v4_projects_issues_search 467 389 467 452 685 405 441 514 16.44% api_v4_projects_languages 58 57 58 59 58 59 58 59 1.43% api_v4_projects_merge_requests 173 172 173 180 166 167 172 171 -0.83% api_v4_projects_merge_requests_merge_request 926 868 926 1017 743 918 907 892 -1.57% api_v4_projects_merge_requests_merge_request_commits 128 128 128 131 123 126 128 127 -1.12% api_v4_projects_merge_requests_merge_request_diffs 109 107 109 114 106 106 108 109 0.50% api_v4_projects_merge_requests_merge_request_discussions 272 250 272 338 260 246 264 281 6.40% api_v4_projects_merge_requests_search 1966 1811 1966 1782 1757 1669 1914 1736 -9.32% api_v4_projects_project 193 187 193 188 188 188 191 188 -1.46% api_v4_projects_project_pipelines 82 80 82 77 76 78 81 77 -4.95% api_v4_projects_project_pipelines_pipeline 101 96 101 95 93 94 99 94 -5.31% api_v4_projects_project_pipelines_pipeline_jobs 1485 1402 1485 1393 1385 1426 1457 1401 -3.83% api_v4_projects_project_services 86 83 86 81 80 83 85 81 -4.72% api_v4_projects_releases 139 139 139 132 129 132 139 131 -5.41% api_v4_projects_repository_branches 86 85 86 87 82 85 86 85 -0.51% api_v4_projects_repository_branches_branch 131 131 131 141 131 132 131 135 2.92% api_v4_projects_repository_branches_search 87 84 87 84 82 84 86 83 -3.12% api_v4_projects_repository_commits 121 121 121 128 122 126 121 125 3.25% api_v4_projects_repository_commits_all 740 794 740 6592 1586 3584 758 3921 417.41% api_v4_projects_repository_commits_commit 120 118 120 118 111 115 119 115 -3.79% api_v4_projects_repository_commits_commit_diff 271 271 271 356 261 276 271 298 9.90% api_v4_projects_repository_compare 121 132 121 126 116 125 124 122 -1.55% api_v4_projects_repository_files_file_blame 18009 26830 18009 22294 21993 21401 20949 21896 4.52% api_v4_projects_repository_files_file_metadata 126 129 126 130 124 141 127 131 3.51% api_v4_projects_repository_files_file_raw 135 134 135 138 132 145 134 138 2.97% api_v4_projects_repository_tags 134 134 134 134 128 143 134 135 0.66% api_v4_projects_repository_tree 138 141 138 138 133 149 139 140 0.95% git_clone 259 467 259 262 421 350 329 344 4.82% git_ls_remote 86 84 86 94 102 114 86 103 20.84% git_pull 141 144 141 148 157 152 142 153 7.60% web_group 233 259 233 239 235 265 242 246 1.92% web_group_issues 225 219 225 201 202 221 223 208 -6.73% web_group_merge_requests 349 342 349 335 340 366 347 347 0.11% web_project 306 347 306 300 313 329 319 314 -1.62% web_project_branches 452 573 452 467 472 525 492 488 -0.90% web_project_branches_search 862 903 862 843 855 917 876 872 -0.51% web_project_commit 1722 1738 1722 1730 1743 1754 1727 1742 0.87% web_project_commits 555 575 555 544 559 560 562 554 -1.32% web_project_file_blame 3389 3545 3389 3038 3074 3466 3441 3193 -7.22% web_project_file_rendered 701 740 701 734 706 702 714 714 0.06% web_project_file_source 677 695 677 686 670 697 683 685 0.27% web_project_files 265 266 265 283 262 267 265 271 2.06% web_project_issue 550 549 550 582 551 555 549 563 2.44% web_project_issues 242 245 242 255 235 239 243 243 -0.10% web_project_issues_search 243 248 243 254 230 235 245 240 -1.98% web_project_merge_request 1805 1807 1805 1780 1784 1786 1806 1783 -1.23% web_project_merge_request_changes 644 658 644 660 639 646 649 648 -0.04% web_project_merge_request_commits 840 934 840 907 826 857 871 863 -0.90% web_project_merge_requests 338 388 338 360 335 340 355 345 -2.69% web_project_merge_requests_search 359 401 359 375 362 354 373 363 -2.63% web_project_pipelines 500 563 500 536 511 514 521 520 -0.16% web_project_pipelines_pipeline 1411 1478 1411 1477 1401 1375 1433 1418 -1.07% web_project_repository_compare 4128 4155 4128 4174 4109 3884 4137 4056 -1.97% web_project_tags 811 858 811 839 774 910 827 841 1.75% - Author Maintainer
Server Performance Metrics: https://snapshots.raintank.io/dashboard/snapshot/5Es5DnycodM3wvexAGoLSfB7wpWhchHC
- Developer
Thanks @john.mcdonnell. Reftable isn't compatible with Praefect. But this (
api_v4_projects_repository_commits_all
) is a read only RPC so it should be okay. So reftable performs worse than freshly packed refs but much much better than non-packed repo. So does this make sense.Was this run with WAL enabled?
CC @pks-gitlab
- Developer
Also speaking about
api_v4_projects_repository_commits_all
, this RPC in the way it is being used in k6 only reads the default branch and lists all commits from it. Reftable only affects the refs part of Git, which is a single operation in this RPC. I'm wondering if there is some reason this is consistently worse in performance.Similar with
api_v4_projects_repository_files_file_blame
, we don't really utilize the reference backend here too.I'm wondering if there is something in our code which is triggering the reference backend for these RPCs.
git_ls_remote
is a good API to check, since it lists all refs, the difference here is very minute though.api_v4_projects_repository_branches
is also a good test for the reference backend. Here too the differennce is minute
Overall I see that we only target read-only RPCs, where reftables is supposed to perform very similar to files backend. It really shines in deletion of refs. I guess such tests are hard to construct under K6 + GPT.
- Developer
@knayakgl @john.mcdonnell This is running with Git v2.46.0, I assume?
Overall I see that we only target read-only RPCs, where reftables is supposed to perform very similar to files backend. It really shines in deletion of refs. I guess such tests are hard to construct under K6 + GPT.
That's true. But unless I misunderstood, the first following line is a fully packed repository whereas the second line is reftables. The TTFB here is way worse, which is surprising.
NAME | RPS | RPS RESULT | TTFB AVG | TTFB P90 | REQ STATUS | RESULT
api_v4_projects_repository_commits_all | 60/s | 55.15/s (>48.00/s) | 439.43ms | 1057.55ms (<200ms) | 100.00% (>99%) | FAILED¹²
api_v4_projects_repository_commits_all | 60/s | 54.8/s (>48.00/s) | 701.00ms | 1669.03ms (<200ms) | 100.00% (>99%) | FAILED¹²
I'd ideally need two things to get started with debugging performance issues like this:
- A reproducible testcase that doesn't involve the whole stack and a full GitLab setup. Ideally, it should only involve setup steps and then the Git command that is being exercised.
- A flame graph of the slow Git command in your environment so that I have a point for comparison against my own environment.
Edited by Patrick Steinhardt
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#19806 (closed)
mentioned in issue gitlab-org/quality/triage-reports#19806 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#19916 (closed)
mentioned in issue gitlab-org/quality/triage-reports#19916 (closed)
- Kassandra Svoboda changed milestone to %17.5
changed milestone to %17.5
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#19998 (closed)
mentioned in issue gitlab-org/quality/triage-reports#19998 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#20132 (closed)
mentioned in issue gitlab-org/quality/triage-reports#20132 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#20247 (closed)
mentioned in issue gitlab-org/quality/triage-reports#20247 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#20329 (closed)
mentioned in issue gitlab-org/quality/triage-reports#20329 (closed)
- Developer
Now that Gitaly runs on Git v2.47 as default, can we revisit this and run the tests? We can also run on Praefect as long as the transaction manager is enabled.
Collapse replies - Developer
@john.mcdonnell Could you please provide an update regarding the current status? Is there anything we can help you with?
- Author Maintainer
Hi @pks-gitlab / @knayakgl We certainly can take a look at rerunning a test suite similar to what we did previously. I think we should be able to reuse a similar test plan as we'd done previously and compare to where we were then. I'll try to get this fitted in this week if we can and get back to you with some results.
Last time we ran this there were a couple of questions around the specific config we used. Perhaps this time I might just ask for some confirmation of the specific set of gitaly/praefect config file changes that need to be applied here just so we can avoid any confusion if I overlook something.
- We're still behind a FF - so we'll need
Feature.enable(:gitaly_new_repo_reftable_backend)
on the rails nodes. - To enable the transaction manager I understand that to mean setting the following on the gitaly nodes.
[transactions] enabled = true
Should there be anything else we need to setup/configure prior to running, just to make sure we try and get it right first time.
1 - We're still behind a FF - so we'll need
- Developer
Should there be anything else we need to setup/configure prior to running
@john.mcdonnell those are the only two requirements.
1 - Author Maintainer
Hi @pks-gitlab / @knayakgl I've posted results of the latest GPT run in #2432 (comment 2248475787)
- Karthik Nayak added gitlab-org#14942 as parent epic
added gitlab-org#14942 as parent epic
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#20445 (closed)
mentioned in issue gitlab-org/quality/triage-reports#20445 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#20582 (closed)
mentioned in issue gitlab-org/quality/triage-reports#20582 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#20750 (closed)
mentioned in issue gitlab-org/quality/triage-reports#20750 (closed)
- 🤖 GitLab Bot 🤖 added Category:Gitaly label
added Category:Gitaly label
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#20830 (closed)
mentioned in issue gitlab-org/quality/triage-reports#20830 (closed)
- 🤖 GitLab Bot 🤖 added devopsdata_access label
added devopsdata_access label
- Author Maintainer
GPT Results for Environment
GitLab Enterprise Edition 17.7.0-pre b4f30955e41 git version 2.47.0 GPT running version [0732d80c](0732d80c547d506eaab337f3aee014efb3a57eb9) The environment was setup as a 3k reference architecture: https://docs.gitlab.com/ee/administration/reference_architectures/3k_users.html Rails environment configured with feature flag `gitaly_new_repo_reftable_backend` enabled. Gitaly configured with transactions enabled
Data Setup
To test this, we setup 2 git repos using
gitlabhq_export_16.0.0.tar.gz
One was created withgitaly_new_repo_reftable_backend
false,One was created with
gitaly_new_repo_reftable_backend
set to true, leading to the repository on disk having the respective repos havin therefstorage = reftable
set. While the second project was created withgitaly_new_repo_reftable_backend
false resulting in it having the 'default' files backend allowing for comparassion to a baseline on the same environmental hardware.For the purpose of the test -- we ran 3 suites with the
gitaly_new_repo_reftable_backend
enabled, and a further 2 with it disabled just to ensure we have a broad range of results, and to help identify any outlying data points. We alternated between running tests between the files/ref backends to help even out any potential for temporary environment instability to cause issues.Results
As noted previously, GPT primarily is composed of a read based requests, so the expectation here would be that the reftables/files backend results should broadly be in alignment.
Having analysed the results of these runs, focusing in on any endpoints that would stress the underlying git repository, there don't appear to be any standout results that either result in a large improvement/regression in response times with almost all results between within a small % delta of each other which is what we are expecting to see.Attached: copy of all results.tar, and in the thread below, a comparison of the AVG/P90/P95 results.
Edited by John McDonnell Collapse replies - Author Maintainer
Time to First Byte Average (ms)
Name files-1 files-2 files-3 files-4 files-5 ref-1 ref-2 ref-3 ref-4 ref-5 files average ref average diff diff % api_v4_groups 1170.64 1116.33 1109.65 1212.67 1156.2 1130.34 1115.12 1112.11 1106.07 1114.92 1153.098 1115.712 -37.4 -3.24% api_v4_groups_group 2619.21 2679.02 2801.71 2770.38 2705.72 2559.2 2929.93 2626.07 2590.11 2940.68 2715.208 2729.198 14.0 0.52% api_v4_groups_group_subgroups 1213.7 1158.03 1151.02 1258.53 1171.61 1148.1 1143.58 1142.91 1140.84 1197.96 1190.578 1154.678 -35.9 -3.02% api_v4_groups_issues 3070.85 3008.63 3058.97 3039.44 3013.24 3050.17 3038.8 3112.14 3024.79 3066.85 3038.226 3058.55 20.3 0.67% api_v4_groups_merge_requests 1075.33 1062.87 1042.86 1146.97 1071.33 1050.05 1046.12 1044.42 1098.76 1093.76 1079.872 1066.622 -13.2 -1.23% api_v4_groups_projects 2517.76 2605.48 2635.97 2601.63 2584.6 2484.19 2571.56 2529.26 2653.86 2703.97 2589.088 2588.568 -0.5 -0.02% api_v4_projects 7210.06 7346.9 7216.33 6437.62 7218 7421.36 7546.55 7015.03 7227.08 7234.43 7085.782 7288.89 203.1 2.87% api_v4_projects_deploy_keys 60.58 60.22 59.06 66.06 59.88 59.59 61.38 60.24 62.46 58.89 61.16 60.512 -0.6 -1.06% api_v4_projects_issues 673.24 626.15 533.86 1412.09 633.9 559.13 603.8 579.96 959.45 559.23 775.848 652.314 -123.5 -15.92% api_v4_projects_issues_issue 746.57 661.57 621.28 690.63 627.43 734.55 706.85 736.37 794.3 751.85 669.496 744.784 75.3 11.25% api_v4_projects_issues_search 844.24 482.25 548.27 508.21 515.54 477.65 485.71 503.89 780.07 527.18 579.702 554.9 -24.8 -4.28% api_v4_projects_languages 61.25 60.91 61.87 60.38 61.95 58.85 61.19 58.73 60.2 60.5 61.272 59.894 -1.4 -2.25% api_v4_projects_merge_requests 167.38 164.64 167.36 172.19 163.98 169.84 166.36 164.91 170.38 163.9 167.11 167.078 0.0 -0.02% api_v4_projects_merge_requests_merge_request 952.39 953.02 955.2 1058.56 962.78 956.7 957.61 944.82 1009.61 961.36 976.39 966.02 -10.4 -1.06% api_v4_projects_merge_requests_merge_request_commits 124.94 121.76 123.93 131.99 121.6 123.48 124.67 122.72 128.24 123.12 124.844 124.446 -0.4 -0.32% api_v4_projects_merge_requests_merge_request_diffs 106.01 103.3 103.88 107.14 103.43 104.11 103.81 107.25 107.66 106.25 104.752 105.816 1.1 1.02% api_v4_projects_merge_requests_merge_request_discussions 600.6 474.24 641.3 811.08 605.69 634.2 614.58 600.57 834.24 604.74 626.582 657.666 31.1 4.96% api_v4_projects_merge_requests_search 2049.02 1977.63 1895.02 1955.03 1967.95 1925.38 1972.52 1897.87 2002.79 1893.1 1968.93 1938.332 -30.6 -1.55% api_v4_projects_project 182.26 181 183.58 192.64 180.5 182.12 184.33 181.09 188.16 183.87 183.996 183.914 -0.1 -0.04% api_v4_projects_project_pipelines 78 75.31 76.16 79.48 75.02 77.17 76.52 76.52 84.15 76.33 76.794 78.138 1.3 1.75% api_v4_projects_project_pipelines_pipeline 93.36 91.79 92.34 95.69 92.68 93.29 92.14 93.53 98.02 92.5 93.172 93.896 0.7 0.78% api_v4_projects_project_pipelines_pipeline_jobs 1382.43 1351.92 1343.03 1443.98 1361.42 1348.03 1377.51 1374.3 1486.63 1400.8 1376.556 1397.454 20.9 1.52% api_v4_projects_project_services 86.43 80.88 81.9 84 79.46 80.55 79.53 81.02 84.38 81.56 82.534 81.408 -1.1 -1.36% api_v4_projects_releases 133.86 130.85 132.57 135.12 130.04 131.43 129.85 128.79 132.67 129.98 132.488 130.544 -1.9 -1.47% api_v4_projects_repository_branches 85.7 83.24 82.26 87.04 81.26 82.9 83.36 83.58 86.27 85.22 83.9 84.266 0.4 0.44% api_v4_projects_repository_branches_branch 115.85 111.31 111.34 115.48 111.91 112.14 110.98 112.8 116.48 113.28 113.178 113.136 0.0 -0.04% api_v4_projects_repository_branches_search 82.89 81.97 81.3 84.86 81.26 82.52 81.5 83.09 85.33 82.14 82.456 82.916 0.5 0.56% api_v4_projects_repository_commits 110.58 109.79 108.69 116.17 109.54 112.33 109.68 113.93 115.58 113.31 110.954 112.966 2.0 1.81% api_v4_projects_repository_commits_all 157.41 153.73 155.14 159.28 154.06 154.28 156.01 154.92 161.63 154.56 155.924 156.28 0.4 0.23% api_v4_projects_repository_commits_commit 102.14 102.61 102.3 106.7 102.52 102.33 103.47 101.67 107.95 101.5 103.254 103.384 0.1 0.13% api_v4_projects_repository_commits_commit_diff 215.52 215.91 220.96 227.93 214.59 213.53 215.02 213.71 224.38 220.01 218.982 217.33 -1.7 -0.75% api_v4_projects_repository_compare 106.79 106.37 104.89 108.83 104.36 104.17 103.85 104.36 109.81 104.08 106.248 105.254 -1.0 -0.94% api_v4_projects_repository_files_file_blame 5760.63 5591.63 6212.91 5646.15 5633.73 5632.16 5663.27 5709.96 6074.81 5684.85 5769.01 5753.01 -16.0 -0.28% api_v4_projects_repository_files_file_metadata 105.71 104.5 106.04 108.81 103.62 103.12 105.72 104.53 112.72 104.07 105.736 106.032 0.3 0.28% api_v4_projects_repository_files_file_raw 112.02 108.1 107.78 113.47 107.15 109.76 109.16 108.13 117.51 108.88 109.704 110.688 1.0 0.90% api_v4_projects_repository_tags 138.42 132.55 132.93 141.06 129.98 131.78 128.51 134.4 135.67 132.19 134.988 132.51 -2.5 -1.84% api_v4_projects_repository_tree 113.3 112.06 111.48 116.78 110.29 112.74 111.05 111.93 117.2 111.81 112.782 112.946 0.2 0.15% api_v4_search_global 1359.67 1029.05 1292.71 1582.01 934.9 1345.53 899.98 1232.11 951.43 1312.09 1239.668 1148.228 -91.4 -7.38% api_v4_search_groups 3700.35 3382.64 4039.69 4844.35 3160.84 4017.06 3631.76 3843.59 3787.32 4231.97 3825.574 3902.34 76.8 2.01% api_v4_search_projects 567.11 517 507.94 766.24 531.27 538.54 498.68 542.43 867.68 556.99 577.912 600.864 23.0 3.97% git_clone 362.02 369.22 446.83 399.29 426.67 355.63 455.65 440.72 446.04 394.34 424.942 30.6 7.76% git_ls_remote 88.17 80.43 78.84 87.79 84.34 84.42 83.77 85.67 87.17 78.97 83.914 84 0.1 0.10% git_pull 113.41 117.82 115.58 121.26 124.47 126.14 126.4 112.46 116.57 112.26 118.508 118.766 0.3 0.22% web_group 282.17 282.4 261.17 314.39 284.62 285.57 276.39 277.16 309.24 280.5 284.95 285.772 0.8 0.29% web_group_issues 282.05 287.36 276.14 287.28 280.61 204.1 200.21 204.2 213.18 211.88 282.688 206.714 -76.0 -26.88% web_group_merge_requests 355.68 362.31 358.07 373.13 352.37 353.66 347.7 350.33 388.34 348.25 360.312 357.656 -2.7 -0.74% web_project 317.27 305.74 302.7 328.25 308.62 317.44 301.57 300.6 314.61 322.73 312.516 311.39 -1.1 -0.36% web_project_branches 457.46 460.45 462.12 478.87 467.24 457.85 466.88 454.49 474.5 468.79 465.228 464.502 -0.7 -0.16% web_project_branches_search 894.33 874.65 894.44 902.32 870.38 877.55 878.58 859.76 928.39 922.92 887.224 893.44 6.2 0.70% web_project_commit 619.35 623.33 610.47 625.05 608.97 609.1 611.84 601.58 618.73 622.48 617.434 612.746 -4.7 -0.76% web_project_commits 508.99 535.1 515.87 580.41 502.71 519.69 513.62 495.18 525.72 517.66 528.616 514.374 -14.2 -2.69% web_project_file_blame 2156.64 2137.94 2126.78 2176.44 2117.3 2194.88 2174.98 2164.08 2197.81 2233.14 2143.02 2192.978 50.0 2.33% web_project_file_rendered 511.7 661.14 503.44 510.83 542.17 532.15 510.21 509.98 664.07 511.77 545.856 545.636 -0.2 -0.04% web_project_file_source 491.88 477.88 485.21 482.34 492.28 479.01 477.53 467.21 488.21 487.45 485.918 479.882 -6.0 -1.24% web_project_files 174.16 163.48 169.97 166.11 166 161.62 168.57 161.26 175.81 166.91 167.944 166.834 -1.1 -0.66% web_project_issue 341.22 346.49 329.95 357.98 330.75 348.27 329.01 327.51 347.29 339.37 341.278 338.29 -3.0 -0.88% web_project_issues 251.9 243.03 250.96 269.26 247.3 254.58 259.98 259.41 267.43 262.81 252.49 260.842 8.4 3.31% web_project_issues_search 245.41 270.32 246.67 231.54 252.67 265.72 248.17 250.27 274.3 250.4 249.322 257.772 8.5 3.39% web_project_merge_request 735.28 717.26 717.94 720.06 732.84 720.4 728.01 697.1 731.05 720.47 724.676 719.406 -5.3 -0.73% web_project_merge_request_changes 507.03 499.87 493.3 504.17 498.2 506.63 498.67 505.52 517.93 505.83 500.514 506.916 6.4 1.28% web_project_merge_request_commits 958.8 956.88 947.93 940.88 963.81 984.7 951.83 952.88 1038.05 960.65 953.66 977.622 24.0 2.51% web_project_merge_requests 348.44 340.24 367.8 353.34 337.89 334.68 340.91 339.98 361.53 341.89 349.542 343.798 -5.7 -1.64% web_project_merge_requests_search 352.29 360.46 350.87 351.14 362.74 351.2 357.29 347.5 375.95 349.39 355.5 356.266 0.8 0.22% web_project_pipelines 376.19 396.44 394.34 382.49 396.81 399.76 390.1 381.97 422.33 392.23 389.254 397.278 8.0 2.06% web_project_pipelines_pipeline 616.11 630.51 604.56 618.2 624.6 615.14 639.48 621.84 647.72 630.53 618.796 630.942 12.1 1.96% web_project_repository_compare 3371.42 3280 3265.3 3359.37 3355.61 3322.83 3304.39 3247.22 3446.47 3442.06 3326.34 3352.594 26.3 0.79% web_project_tags 739.22 733.33 743.88 747.74 752.92 736.26 738.75 737.47 780.22 755.36 743.418 749.612 6.2 0.83% web_search_global 170.38 167.52 106.57 181.47 80.48 90.72 189.69 170.92 103.78 174.11 141.284 145.844 4.6 3.23% web_search_groups 75.21 64.9 118.95 72.16 122.72 100.72 65.49 71.83 125.73 83.19 90.788 89.392 -1.4 -1.54% web_search_projects 108.39 108.99 64.48 117.27 74.47 90.36 118.04 123.38 61.26 120.99 94.72 102.806 8.1 8.54% Time to First Byte P90 (ms)
Name files-1 files-2 files-3 files-4 files-5 ref-1 ref-2 ref-3 ref-4 ref-5 files average ref average diff diff % api_v4_groups 1306.78 1260.66 1241.53 1440.08 1314.17 1275.92 1265 1257.63 1248.89 1260.12 1312.644 1261.512 -51.1 -3.90% api_v4_groups_group 3338.84 4266.53 4327.04 3775.16 4494.88 3071.86 4367.78 4052.87 3254.66 4478.95 4040.49 3845.224 -195.3 -4.83% api_v4_groups_group_subgroups 1376.66 1310.54 1295.97 1517.36 1325.04 1301.23 1306.37 1306.63 1298.34 1343.85 1365.114 1311.284 -53.8 -3.94% api_v4_groups_issues 5290.21 4666.14 4592.88 4688.78 4690.89 4621.65 4731.34 4902.27 4614.05 4587.55 4785.78 4691.372 -94.4 -1.97% api_v4_groups_merge_requests 1205.26 1193.38 1181.81 1354.75 1208.7 1187.02 1184.68 1183.64 1272.41 1222 1228.78 1209.95 -18.8 -1.53% api_v4_groups_projects 3210.5 3710.95 4058.08 3437.79 3520.56 3063.15 4183.94 3273.15 3252.48 4642.92 3587.576 3683.128 95.6 2.66% api_v4_projects 8705.44 8878.45 8633.69 12752.26 8678.86 8936.39 8951.74 8401.38 8888.17 8687.83 9529.74 8773.102 -756.6 -7.94% api_v4_projects_deploy_keys 64.86 65.15 64.3 72.27 64.24 64 66.11 64.58 70.4 63.53 66.164 65.724 -0.4 -0.67% api_v4_projects_issues 965.44 859.28 736.11 1716.83 858.29 778.94 826.45 857.81 1164.18 771.84 1027.19 879.844 -147.3 -14.34% api_v4_projects_issues_issue 1056.08 933.81 913.83 954.43 952.26 1023.61 974.44 979.28 1026.35 1059.8 962.082 1012.696 50.6 5.26% api_v4_projects_issues_search 1145.7 695.47 828.33 730.99 730.4 779.49 703.46 732.55 1049.34 817.38 826.178 816.444 -9.7 -1.18% api_v4_projects_languages 66.45 64.13 65.71 67.58 64.85 63.06 63.92 63.23 67.69 64.41 65.744 64.462 -1.3 -1.95% api_v4_projects_merge_requests 188.95 185.59 188.93 198.69 184.61 189.32 185.85 186.06 197 185.22 189.354 188.69 -0.7 -0.35% api_v4_projects_merge_requests_merge_request 1095.76 1091.64 1105.16 1327.08 1100.78 1106.18 1098.51 1078.48 1179.52 1096.23 1144.084 1111.784 -32.3 -2.82% api_v4_projects_merge_requests_merge_request_commits 139.47 135.59 137.9 147.1 135.6 137.56 139.65 136.5 143.36 137.01 139.132 138.816 -0.3 -0.23% api_v4_projects_merge_requests_merge_request_diffs 117.62 113.69 115.52 119.65 113.85 113.92 114.34 117.84 119.62 115.1 116.066 116.164 0.1 0.08% api_v4_projects_merge_requests_merge_request_discussions 870.34 683.24 917.24 1054.05 877.96 912.87 893.17 893.91 1069.57 907.75 880.566 935.454 54.9 6.23% api_v4_projects_merge_requests_search 3906.12 4239.05 3144.74 3786.13 4344.78 3557.06 4730.27 3190.27 4219.8 3356.94 3884.164 3810.868 -73.3 -1.89% api_v4_projects_project 208.88 206.89 208.51 224.35 204.44 207.14 208.29 205.22 218.41 206.06 210.614 209.024 -1.6 -0.75% api_v4_projects_project_pipelines 86.61 84.47 84.74 89.82 83.26 84.61 85.44 85.45 93.56 84.33 85.78 86.678 0.9 1.05% api_v4_projects_project_pipelines_pipeline 104.7 103.24 102.46 108.15 102.19 102.18 102.16 103.2 112.33 102.77 104.148 104.528 0.4 0.36% api_v4_projects_project_pipelines_pipeline_jobs 1612.28 1595.23 1588.09 1723.02 1615.73 1585.66 1622.53 1601.86 1851 1646.59 1626.87 1661.528 34.7 2.13% api_v4_projects_project_services 94.73 90.67 90.99 96.32 89.13 90.46 89.82 91.36 96.61 94.39 92.368 92.528 0.2 0.17% api_v4_projects_releases 154.78 143.55 144.08 151.96 143.21 143.94 142.89 142.34 151.84 147.57 147.516 145.716 -1.8 -1.22% api_v4_projects_repository_branches 95.65 92.68 92 99.33 91.35 92.49 94.12 93.27 98.53 98.13 94.202 95.308 1.1 1.17% api_v4_projects_repository_branches_branch 129.59 124.2 123.19 129.91 124.06 125.41 124.3 124.49 133.45 127.12 126.19 126.954 0.8 0.61% api_v4_projects_repository_branches_search 93.31 92.48 91.48 95.65 91.03 92.91 91.65 93.15 97.23 92.81 92.79 93.55 0.8 0.82% api_v4_projects_repository_commits 123.35 123.1 121.5 129.41 121.7 125.19 122.45 124.36 130.81 124.74 123.812 125.51 1.7 1.37% api_v4_projects_repository_commits_all 170.83 167.87 170.36 174.33 167.14 168.54 169.41 168.83 177.78 169.32 170.106 170.776 0.7 0.39% api_v4_projects_repository_commits_commit 115.43 116.34 114.91 120.62 115.71 114.98 117.09 114.41 124.19 114.19 116.602 116.972 0.4 0.32% api_v4_projects_repository_commits_commit_diff 238.17 238.2 245.59 258.81 237.18 234.92 238.91 236.22 253.22 247.36 243.59 242.126 -1.5 -0.60% api_v4_projects_repository_compare 116.53 116.85 113.88 120.49 113.21 113.63 114.27 112.28 121.23 113.6 116.192 115.002 -1.2 -1.02% api_v4_projects_repository_files_file_blame 9539.41 9791.7 10536.69 8158.39 9877.83 10079.47 10707.88 9766.22 13182.14 8381.33 9580.804 10423.408 842.6 8.79% api_v4_projects_repository_files_file_metadata 117.23 115.08 116.31 121.47 114.14 113.37 116.01 114.86 125.94 114.04 116.846 116.844 0.0 0.00% api_v4_projects_repository_files_file_raw 123.22 119.75 119.16 128.21 117.96 121.21 120.16 119.2 131.38 120.31 121.66 122.452 0.8 0.65% api_v4_projects_repository_tags 147.75 141.12 140.9 150.79 139.04 141.27 138.57 142.57 148 141.13 143.92 142.308 -1.6 -1.12% api_v4_projects_repository_tree 127.16 124.82 123.84 131.49 121.85 124.11 123.3 124.34 132.17 123.45 125.832 125.474 -0.4 -0.28% api_v4_search_global 3048.42 1479.77 5719.75 6867.32 1054.7 4379.56 912.81 7908.21 1026.85 7810.24 3633.992 4407.534 773.5 21.29% api_v4_search_groups 9533.66 9775.49 10804.79 12486.45 8906.52 10528.85 9441.67 11395 9917.51 10280.35 10301.382 10312.676 11.3 0.11% api_v4_search_projects 1039.56 964.2 942.99 1472.65 982.77 993.69 931.54 1009.36 1679.12 1035.28 1080.434 1129.798 49.4 4.57% git_clone 1279.89 1329.21 1672 1831.84 1408.39 1546.47 1476.17 1711.34 1431.52 1513.98 1504.266 1535.896 31.6 2.10% git_ls_remote 91.19 87.86 87.63 101.21 99.31 93.08 90.76 89.96 97.29 88.47 93.44 91.912 -1.5 -1.64% git_pull 138.22 136.97 135.96 187.7 145.84 143.79 145.99 137.51 143.01 136.29 148.938 141.318 -7.6 -5.12% web_group 339.65 349.87 332.83 387.79 353.43 342.12 352.94 338.59 403.88 341.14 352.714 355.734 3.0 0.86% web_group_issues 331.78 330.55 321.26 347.85 341.06 223.41 224.51 219.73 236.96 226.72 334.5 226.266 -108.2 -32.36% web_group_merge_requests 368.22 369.88 372.67 394.59 368.22 369.95 360.85 359.68 388.83 371.71 374.716 370.204 -4.5 -1.20% web_project 333.08 328.73 322.2 348.26 320.32 337.12 320.83 323.64 331.57 335.83 330.518 329.798 -0.7 -0.22% web_project_branches 507.29 494.46 487.08 520.36 508.67 494.45 520.57 482.99 518.54 501.52 503.572 503.614 0.0 0.01% web_project_branches_search 1358.38 1159.52 1006.44 1213.73 1112.68 1180.36 1116.75 951.97 1222.69 1334.49 1170.15 1161.252 -8.9 -0.76% web_project_commit 1728.85 1729.94 1721.95 1747.7 1717.2 1733.45 1727.63 1703.21 1766.77 1742.72 1729.128 1734.756 5.6 0.33% web_project_commits 552.47 568.39 546.01 619.79 533.41 553.05 548.63 534.7 578.12 558.51 564.014 554.602 -9.4 -1.67% web_project_file_blame 2921.76 2883.75 2801.37 2942.51 2877.06 2958.73 2921.42 2927.2 2960.62 3042.49 2885.29 2962.092 76.8 2.66% web_project_file_rendered 672.35 886.21 694.79 701.82 781.63 751.44 632.58 686.66 888.09 695.43 747.36 730.84 -16.5 -2.21% web_project_file_source 608.48 592.11 599.83 596.61 597.06 601.17 597.05 591.69 604.61 599.85 598.818 598.874 0.1 0.01% web_project_files 256.22 256.13 251.16 252.17 256.5 259.5 256.07 247.83 278.62 254.22 254.436 259.248 4.8 1.89% web_project_issue 601.52 622.44 620.29 624.34 607.99 620.36 605.44 593.36 627.46 617.58 615.316 612.84 -2.5 -0.40% web_project_issues 253.72 247.33 246.58 255.35 244.27 251.03 248.67 242.39 265.84 248.88 249.45 251.362 1.9 0.77% web_project_issues_search 248.75 249.02 245.04 237.3 243.55 250.6 245.76 243.54 266.85 245.58 244.732 250.466 5.7 2.34% web_project_merge_request 1832.75 1834.14 1821.55 1815.46 1840.74 1848.25 1826.87 1810.55 1852.69 1835.73 1828.928 1834.818 5.9 0.32% web_project_merge_request_changes 651.47 644.7 639.29 653.65 654.59 654.42 650.39 651.54 669.97 655.56 648.74 656.376 7.6 1.18% web_project_merge_request_commits 1036.58 1042.3 1007.43 1028.59 1029.24 1042.7 1046.12 1017.09 1168.17 1030.41 1028.828 1060.898 32.1 3.12% web_project_merge_requests 355.19 358.14 364.81 359.53 360.94 354.74 362.23 352.5 399.91 362.55 359.722 366.386 6.7 1.85% web_project_merge_requests_search 383.98 385.79 382.13 381.41 392.08 386.68 383.83 376.93 424.89 382.45 385.078 390.956 5.9 1.53% web_project_pipelines 539.18 549.31 542.79 537.39 558.36 546.11 549.4 530.61 595.24 533.91 545.406 551.054 5.6 1.04% web_project_pipelines_pipeline 1357.56 1408.53 1396.74 1401.19 1406.87 1381.49 1397.43 1370.82 1461.73 1382.34 1394.178 1398.762 4.6 0.33% web_project_repository_compare 4121.7 4129.5 4068.18 4195.45 4139.07 4126.73 4141.55 4105.64 4255.13 4259.18 4130.78 4177.646 46.9 1.13% web_project_tags 819.86 796.45 793 810.31 842.36 798.14 800.13 798.45 858.21 812.77 812.396 813.54 1.1 0.14% web_search_global 601.08 554.9 172.22 561.45 119.38 138.8 613.17 529.02 143.6 558.32 401.806 396.582 -5.2 -1.30% web_search_groups 88.78 83.72 108.6 86.66 95.48 91.65 87.02 87.99 114.52 99.15 92.648 96.066 3.4 3.69% web_search_projects 93.88 113.21 80.71 118.51 87.51 87.93 151.76 109.53 86.65 145.81 98.764 116.336 17.6 17.79% Time to First Byte P95 (ms)
Name files-1 files-2 files-3 files-4 files-5 ref-1 ref-2 ref-3 ref-4 ref-5 files average ref average diff diff % api_v4_groups 1397.94 1322.47 1289.74 1539.8 1402.88 1353.4 1308.07 1317.29 1301.26 1309.23 1390.566 1317.85 -72.7 -5.23% api_v4_groups_group 3958.68 4819.32 4690.04 4655.83 4912.83 3522.21 4719.06 4703.98 3828.37 4867.97 4607.34 4328.318 -279.0 -6.06% api_v4_groups_group_subgroups 1458.97 1359.58 1346.29 1629.4 1363.83 1347.18 1348.61 1357.42 1333.04 1414.95 1431.614 1360.24 -71.4 -4.99% api_v4_groups_issues 5688.35 4769.74 4733.96 4842.31 4874.54 4754.34 4859.7 5072.32 4786.92 4753.93 4981.78 4845.442 -136.3 -2.74% api_v4_groups_merge_requests 1287.27 1272.11 1247.57 1505.33 1294.48 1293.16 1248.52 1261.1 1335.07 1316.19 1321.352 1290.808 -30.5 -2.31% api_v4_groups_projects 3703.83 4797.02 4943.88 3970.53 4552.11 3564.59 4668.61 3841.32 3884.16 5148.85 4393.474 4221.506 -172.0 -3.91% api_v4_projects 8952.98 9205.33 9107.58 14735.03 9057.66 10074.78 10406.68 8744.59 9182.99 8925.15 10211.716 9466.838 -744.9 -7.29% api_v4_projects_deploy_keys 68.48 69.45 68.18 77.93 68.16 67.17 70.4 70.28 74.31 67.28 70.44 69.888 -0.6 -0.78% api_v4_projects_issues 1026.66 915.02 793.53 1785.54 906.58 843.49 887.81 934.71 1233.73 829.41 1085.466 945.83 -139.6 -12.86% api_v4_projects_issues_issue 1551.93 1207.53 1086.24 1064.19 1252.55 1538.76 1414 1385.94 1316.29 1594.11 1232.488 1449.82 217.3 17.63% api_v4_projects_issues_search 1321.51 786.52 916.06 830.49 813.84 911.25 802.44 826.61 1189.4 939.55 933.684 933.85 0.2 0.02% api_v4_projects_languages 73.84 70.15 74.26 72.01 72.33 67.62 69.31 67.75 71.47 70.77 72.518 69.384 -3.1 -4.32% api_v4_projects_merge_requests 198.67 194.2 198.59 209.62 194.1 200.53 194.26 195.39 207.9 194.33 199.036 198.482 -0.6 -0.28% api_v4_projects_merge_requests_merge_request 1188.64 1185.58 1186.73 1577.72 1178.37 1193.06 1201.62 1182.51 1269.04 1189.86 1263.408 1207.218 -56.2 -4.45% api_v4_projects_merge_requests_merge_request_commits 148.06 144.1 150.6 162.48 144.07 145.69 150.83 145.31 151.78 146.13 149.862 147.948 -1.9 -1.28% api_v4_projects_merge_requests_merge_request_diffs 122.7 119.18 120.17 124.2 119.01 118.98 119.18 125.93 124.56 120.7 121.052 121.87 0.8 0.68% api_v4_projects_merge_requests_merge_request_discussions 940.09 754.06 969.5 1094.35 931.4 968.15 951.12 955.31 1118.01 965.77 937.88 991.672 53.8 5.74% api_v4_projects_merge_requests_search 4797.18 4801.59 3429.1 4317.26 4946.8 4069.06 5182.69 3566.46 4708.7 3781.44 4458.386 4261.67 -196.7 -4.41% api_v4_projects_project 218.55 217.53 224.97 239.58 218.72 220.54 220.93 220.05 229.5 221.76 223.87 222.556 -1.3 -0.59% api_v4_projects_project_pipelines 92.67 88.51 89.39 94.58 87.61 89.78 90 90.75 99.83 89.4 90.552 91.952 1.4 1.55% api_v4_projects_project_pipelines_pipeline 109.54 108.49 107.99 113.4 108.72 109.2 107.69 110.38 117.48 109.05 109.628 110.76 1.1 1.03% api_v4_projects_project_pipelines_pipeline_jobs 1707.94 1671.74 1659.5 1822.22 1745.93 1670.15 1764.42 1711.05 2081.32 1779.02 1721.466 1801.192 79.7 4.63% api_v4_projects_project_services 101.28 96.58 98.17 101.81 95.79 98.49 95.57 97.99 102.15 99.94 98.726 98.828 0.1 0.10% api_v4_projects_releases 163.22 151.97 152.17 161.7 151.2 154.26 152.77 148.82 160.87 155.58 156.052 154.46 -1.6 -1.02% api_v4_projects_repository_branches 101.29 98.73 97.54 106.79 96.63 98.91 100.42 98.85 104.32 104.59 100.196 101.418 1.2 1.22% api_v4_projects_repository_branches_branch 135.95 129.55 128.78 135.42 129.8 130.6 129.5 130.01 139.79 133.38 131.9 132.656 0.8 0.57% api_v4_projects_repository_branches_search 98.6 97.91 96.94 100.94 96.04 98.51 96.52 99.04 102.67 97.85 98.086 98.918 0.8 0.85% api_v4_projects_repository_commits 129.87 129.26 127.18 136.73 128.38 131.72 128.71 131.26 137.12 131.6 130.284 132.082 1.8 1.38% api_v4_projects_repository_commits_all 177.12 174.14 177.97 181.31 173.55 174.06 177.17 177.61 186.12 176.33 176.818 178.258 1.4 0.81% api_v4_projects_repository_commits_commit 121.59 122.96 120.61 127.91 122.67 120.58 124.26 119.68 130.79 120.93 123.148 123.248 0.1 0.08% api_v4_projects_repository_commits_commit_diff 250.26 253.48 276.02 298.13 253.14 246.46 253.71 249.03 274.99 272.68 266.206 259.374 -6.8 -2.57% api_v4_projects_repository_compare 127.61 127.43 123.37 130.23 127.6 122.95 122.22 123.22 131.4 123.85 127.248 124.728 -2.5 -1.98% api_v4_projects_repository_files_file_blame 10104.99 10504.8 10875.08 8472.14 10288.92 10635.84 12457.56 10745.06 13718.46 9978.15 10049.186 11507.014 1457.8 14.51% api_v4_projects_repository_files_file_metadata 123.01 122 128.91 127.18 120.42 120.01 124.35 121.46 135.64 121.77 124.304 124.646 0.3 0.28% api_v4_projects_repository_files_file_raw 129.38 125.81 124.95 134.59 124.54 127.82 126.6 125.12 137.76 126.42 127.854 128.744 0.9 0.70% api_v4_projects_repository_tags 161.13 148.99 149.26 164.05 147.32 151.22 145.9 154.34 155.91 152.99 154.15 152.072 -2.1 -1.35% api_v4_projects_repository_tree 133.26 130.67 129.83 138.19 128.81 130.42 129.8 130.19 137.82 129.14 132.152 131.474 -0.7 -0.51% api_v4_search_global 9031.44 8300.48 9948.59 10804.51 8580.77 9713.47 8340.62 8703.43 8165.42 8865.7 9333.158 8757.728 -575.4 -6.17% api_v4_search_groups 12610.32 12566.76 15144.89 14246.82 10781.11 11990.29 11220.91 15460.29 11381.23 12342.31 13069.98 12479.006 -591.0 -4.52% api_v4_search_projects 1334.68 1192.82 1151.73 1873.86 1203.37 1246.1 1132.82 1270.03 2148.38 1307.25 1351.292 1420.916 69.6 5.15% git_clone 1658.6 1609.32 1840.14 1957.44 1698.25 1670.66 1623.02 1974.97 1795.21 1764.22 1752.75 1765.616 12.9 0.73% git_ls_remote 114.57 104.95 105.94 116.15 114.18 112.35 106.47 110.24 116.03 110.23 111.158 111.064 -0.1 -0.08% git_pull 163.54 161.36 157.75 198.23 176.25 188.96 185.31 160.91 158.4 162.86 171.426 171.288 -0.1 -0.08% web_group 525.27 432.6 347.54 698.25 461.96 655.63 390.2 409.37 605.65 645.49 493.124 541.268 48.1 9.76% web_group_issues 354.51 359.42 364.82 374.46 370.8 236.67 235.36 232.72 261.42 243.67 364.802 241.968 -122.8 -33.67% web_group_merge_requests 570.02 808.23 830.79 802.14 817.05 734.39 754.03 782.78 843.68 431.52 765.646 709.28 -56.4 -7.36% web_project 587.59 507.84 516.65 555.25 520.04 579.11 552.06 465.62 359.3 620.96 537.474 515.41 -22.1 -4.11% web_project_branches 579.64 575.09 568.21 581.6 585.36 570.62 668.13 565.23 608.5 606.62 577.98 603.82 25.8 4.47% web_project_branches_search 1683.57 1490.31 1509.05 1581.92 1460.91 1501.69 1440.41 1598.66 1536.49 1655.02 1545.152 1546.454 1.3 0.08% web_project_commit 1853.14 1845.72 1817.24 1814.76 1772.73 1762.67 1756.79 1775 1822.17 1853.82 1820.718 1794.09 -26.6 -1.46% web_project_commits 582.87 600.56 570.7 682.35 554.39 573.66 561.45 544.56 592.13 582.87 598.174 570.934 -27.2 -4.55% web_project_file_blame 2987.86 2991.09 2990.61 3106.43 2993.84 3421.83 3036.96 3396.21 3151.76 4073.39 3013.966 3416.03 402.1 13.34% web_project_file_rendered 868.51 1467.83 1070.85 1052.16 1283.77 1324.25 1266.74 1334.34 1563.31 864.99 1148.624 1270.726 122.1 10.63% web_project_file_source 649.03 610.33 620.39 615.41 678.91 622.99 623.54 611.89 625.31 633.39 634.814 623.424 -11.4 -1.79% web_project_files 287 286.49 310.96 280.67 295.12 285.92 305.67 281.72 333.78 291.08 292.048 299.634 7.6 2.60% web_project_issue 651.92 678.64 652.63 698.09 647.24 664.8 633.26 639.23 697.07 648.51 665.704 656.574 -9.1 -1.37% web_project_issues 272.23 261.49 263.17 272.3 256.84 263.49 262.06 267.9 289.06 274.89 265.206 271.48 6.3 2.37% web_project_issues_search 264.94 267.7 282.9 251.46 262.04 273.98 260.74 259.03 290.31 266.43 265.808 270.098 4.3 1.61% web_project_merge_request 1924.28 1885.64 1894.1 1905.58 1910.52 1884.94 1881.63 1863.96 1969.76 1891.67 1904.024 1898.392 -5.6 -0.30% web_project_merge_request_changes 692.37 685.71 683.11 683.27 678.71 681.73 681.81 703.15 715.7 696.18 684.634 695.714 11.1 1.62% web_project_merge_request_commits 1202.81 1398.32 1295.59 1079.14 1153.9 1844.91 1410.26 1138.37 1523.29 1111.42 1225.952 1405.65 179.7 14.66% web_project_merge_requests 401.9 389.19 426.54 397.51 392.73 387.8 392.92 378.18 419.9 397.38 401.574 395.236 -6.3 -1.58% web_project_merge_requests_search 397.38 404.24 395.9 399.59 412.19 396.8 400.53 391.51 441.34 391.29 401.86 404.294 2.4 0.61% web_project_pipelines 615.87 654.41 620.79 630.68 649.78 660.84 634.47 621.25 703.03 640.31 634.306 651.98 17.7 2.79% web_project_pipelines_pipeline 1518.17 1541.98 1485.88 1510.23 1530.11 1519.75 1543.44 1513.7 1635.51 1502.81 1517.274 1543.042 25.8 1.70% web_project_repository_compare 4235.59 4218.06 4202.24 4277.51 4232.53 4268.16 4193.41 4178.36 4460.99 4301.33 4233.186 4280.45 47.3 1.12% web_project_tags 973.69 976.49 1069.23 999.48 1182.64 1134.97 1138.63 1006.04 1146.1 1105.23 1040.306 1106.194 65.9 6.33% web_search_global 936.18 840.17 368.66 831.3 321.88 338.94 931.31 888.21 398.12 840.6 659.638 679.436 19.8 3.00% web_search_groups 105.38 95.83 607.55 95.79 616.33 489.75 101.68 108.35 515.35 132.6 304.176 269.546 -34.6 -11.38% web_search_projects 483.61 588.48 88.69 591.35 110.71 478.56 597.31 568.28 99.67 609.53 372.568 470.67 98.1 26.33% - Author Maintainer
- Developer
Considering this is read only. This seems good. Shows there is no regression with using reftables.
The issue with
api_v4_projects_repository_commits_all
is also now no longer present. This helps a bunch. Thanks!
- Kassandra Svoboda mentioned in epic gitlab-org/quality#99
mentioned in epic gitlab-org/quality#99
- service-epic-status-automation mentioned in epic gitlab-org/quality&99
mentioned in epic gitlab-org/quality&99
- service-epic-status-automation mentioned in epic gitlab-org/quality&96
mentioned in epic gitlab-org/quality&96