Add shell benchmarks
What does this MR do?
Adds shell benchmarks.
The setup (runner settings, job response) have been chosen to touch upon the most common code paths (I used code coverage to determine).
Why was this MR needed?
With trace improvements recently merged, we've been looking at CPU and memory profiles.
Now that trace performance is no longer an issue, it's probably a good idea to add baseline benchmarks for other hot code paths that have the potential to become issues or we believe we can further improve.
One hot code path is the shell implementations. Every single job requires the script for various stages to be generated.
What's the best way to test this MR?
N/A - it's just benchmarks.
What are the relevant issue numbers?
Closes #27865 (closed)
Benchmarks
% go test -run XXX -bench=BenchmarkScriptStage --benchtime=5s ./shells
goos: darwin
goarch: amd64
pkg: gitlab.com/gitlab-org/gitlab-runner/shells
cpu: Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
BenchmarkScriptStage/sh-prepare_script-16 1000000 5873 ns/op 18.22 MB/s 8803 B/op 55 allocs/op
BenchmarkScriptStage/sh-get_sources-16 79672 76230 ns/op 36.42 MB/s 59214 B/op 702 allocs/op
BenchmarkScriptStage/sh-restore_cache-16 89497 64728 ns/op 38.51 MB/s 53666 B/op 522 allocs/op
BenchmarkScriptStage/sh-download_artifacts-16 129776 46851 ns/op 38.68 MB/s 41401 B/op 387 allocs/op
BenchmarkScriptStage/sh-after_script-16 148039 41013 ns/op 39.16 MB/s 34285 B/op 344 allocs/op
BenchmarkScriptStage/sh-archive_cache-16 100034 60527 ns/op 33.95 MB/s 47605 B/op 482 allocs/op
BenchmarkScriptStage/sh-archive_cache_on_failure-16 101766 56596 ns/op 36.31 MB/s 47607 B/op 482 allocs/op
BenchmarkScriptStage/sh-upload_artifacts_on_success-16 123105 46212 ns/op 39.60 MB/s 41890 B/op 397 allocs/op
BenchmarkScriptStage/sh-upload_artifacts_on_failure-16 130189 46477 ns/op 39.37 MB/s 41890 B/op 397 allocs/op
BenchmarkScriptStage/sh-cleanup_file_variables-16 981684 6132 ns/op 16.96 MB/s 8811 B/op 65 allocs/op
BenchmarkScriptStage/sh-step_release-16 144720 46590 ns/op 35.78 MB/s 35126 B/op 360 allocs/op
BenchmarkScriptStage/bash-prepare_script-16 957662 6153 ns/op 17.72 MB/s 8803 B/op 55 allocs/op
BenchmarkScriptStage/bash-get_sources-16 65799 83050 ns/op 33.45 MB/s 59214 B/op 702 allocs/op
BenchmarkScriptStage/bash-restore_cache-16 94116 66154 ns/op 37.72 MB/s 53666 B/op 522 allocs/op
BenchmarkScriptStage/bash-download_artifacts-16 131709 48231 ns/op 37.61 MB/s 41401 B/op 387 allocs/op
BenchmarkScriptStage/bash-after_script-16 134527 43380 ns/op 37.07 MB/s 34285 B/op 344 allocs/op
BenchmarkScriptStage/bash-archive_cache-16 94912 59887 ns/op 34.35 MB/s 47606 B/op 482 allocs/op
BenchmarkScriptStage/bash-archive_cache_on_failure-16 102358 56542 ns/op 36.38 MB/s 47606 B/op 482 allocs/op
BenchmarkScriptStage/bash-upload_artifacts_on_success-16 130976 46399 ns/op 39.48 MB/s 41890 B/op 397 allocs/op
BenchmarkScriptStage/bash-upload_artifacts_on_failure-16 130428 46060 ns/op 39.77 MB/s 41890 B/op 397 allocs/op
BenchmarkScriptStage/bash-cleanup_file_variables-16 983510 6179 ns/op 17.15 MB/s 8811 B/op 65 allocs/op
BenchmarkScriptStage/bash-step_release-16 139119 41218 ns/op 40.49 MB/s 35127 B/op 360 allocs/op
BenchmarkScriptStage/pwsh-prepare_script-16 1673475 3597 ns/op 23.63 MB/s 6921 B/op 35 allocs/op
BenchmarkScriptStage/pwsh-get_sources-16 91208 65765 ns/op 103.99 MB/s 76882 B/op 630 allocs/op
BenchmarkScriptStage/pwsh-restore_cache-16 122905 48943 ns/op 96.05 MB/s 63961 B/op 486 allocs/op
BenchmarkScriptStage/pwsh-download_artifacts-16 159080 35342 ns/op 102.68 MB/s 38863 B/op 360 allocs/op
BenchmarkScriptStage/pwsh-after_script-16 186247 31266 ns/op 104.39 MB/s 35432 B/op 324 allocs/op
BenchmarkScriptStage/pwsh-archive_cache-16 142920 46136 ns/op 87.05 MB/s 42212 B/op 451 allocs/op
BenchmarkScriptStage/pwsh-archive_cache_on_failure-16 134970 44577 ns/op 90.09 MB/s 42212 B/op 451 allocs/op
BenchmarkScriptStage/pwsh-upload_artifacts_on_success-16 161738 38780 ns/op 94.04 MB/s 39335 B/op 365 allocs/op
BenchmarkScriptStage/pwsh-upload_artifacts_on_failure-16 152370 38154 ns/op 95.59 MB/s 39336 B/op 365 allocs/op
BenchmarkScriptStage/pwsh-cleanup_file_variables-16 945076 5640 ns/op 68.97 MB/s 9636 B/op 52 allocs/op
BenchmarkScriptStage/pwsh-step_release-16 189794 33264 ns/op 103.44 MB/s 35889 B/op 343 allocs/op
BenchmarkScriptStage/powershell-prepare_script-16 1652925 3574 ns/op 15.95 MB/s 6809 B/op 34 allocs/op
BenchmarkScriptStage/powershell-get_sources-16 89972 67359 ns/op 103.59 MB/s 79744 B/op 637 allocs/op
BenchmarkScriptStage/powershell-restore_cache-16 114195 53018 ns/op 90.27 MB/s 64074 B/op 487 allocs/op
BenchmarkScriptStage/powershell-download_artifacts-16 166354 35041 ns/op 105.05 MB/s 38975 B/op 361 allocs/op
BenchmarkScriptStage/powershell-after_script-16 198408 31175 ns/op 106.05 MB/s 35545 B/op 325 allocs/op
BenchmarkScriptStage/powershell-archive_cache-16 139818 44662 ns/op 91.38 MB/s 42325 B/op 452 allocs/op
BenchmarkScriptStage/powershell-archive_cache_on_failure-16 139326 42907 ns/op 95.11 MB/s 42325 B/op 452 allocs/op
BenchmarkScriptStage/powershell-upload_artifacts_on_success-16 158066 35728 ns/op 103.53 MB/s 39465 B/op 366 allocs/op
BenchmarkScriptStage/powershell-upload_artifacts_on_failure-16 162217 36418 ns/op 101.57 MB/s 39465 B/op 366 allocs/op
BenchmarkScriptStage/powershell-cleanup_file_variables-16 1000000 5799 ns/op 63.29 MB/s 9556 B/op 52 allocs/op
BenchmarkScriptStage/powershell-step_release-16 171282 33595 ns/op 103.88 MB/s 37924 B/op 344 allocs/op
BenchmarkScriptStage/cmd-prepare_script-16 1362862 4350 ns/op 47.59 MB/s 7696 B/op 39 allocs/op
BenchmarkScriptStage/cmd-get_sources-16 132835 45250 ns/op 70.01 MB/s 40216 B/op 405 allocs/op
BenchmarkScriptStage/cmd-restore_cache-16 185707 32849 ns/op 68.89 MB/s 27794 B/op 287 allocs/op
BenchmarkScriptStage/cmd-download_artifacts-16 245910 25858 ns/op 64.51 MB/s 22842 B/op 196 allocs/op
BenchmarkScriptStage/cmd-after_script-16 267549 23305 ns/op 62.39 MB/s 17585 B/op 163 allocs/op
BenchmarkScriptStage/cmd-archive_cache-16 181905 34287 ns/op 53.90 MB/s 26328 B/op 264 allocs/op
BenchmarkScriptStage/cmd-archive_cache_on_failure-16 186092 33375 ns/op 55.37 MB/s 26328 B/op 264 allocs/op
BenchmarkScriptStage/cmd-upload_artifacts_on_success-16 231283 25435 ns/op 66.21 MB/s 23219 B/op 199 allocs/op
BenchmarkScriptStage/cmd-upload_artifacts_on_failure-16 240145 25614 ns/op 65.74 MB/s 23219 B/op 199 allocs/op
BenchmarkScriptStage/cmd-cleanup_file_variables-16 1349767 4345 ns/op 28.77 MB/s 7024 B/op 41 allocs/op
BenchmarkScriptStage/cmd-step_release-16 264811 22765 ns/op 70.02 MB/s 21584 B/op 178 allocs/op
Edited by Arran Walker