Contrib script to compare benchmark results
This MR adds two new scripts:
-
contrib/bench/bench_compare.sh
: Compare results of two benchmark runs, and report the biggest changes in both time elapsed and speed of execution, by percent, sorted by the change amount.- By default, reports only changes greater than 10%. This is configurable with the
--threshold
option. - By default, reports only changes greater than 10%. This is configurable with the
--threshold x
option. - Colour output is enabled by default in interactive terminals, disabled otherwise, controlled by both
--colour
and--color
flags to allow for transatlantic flexibility ;) Accepts "on", "off" (and common synonyms like "true", "false"), and "auto".
- By default, reports only changes greater than 10%. This is configurable with the
-
contrib/bench/bench_compare_branch.sh
: Check out, build and run benchmarks on two branches, and compare their results.- If one branch is specified, it compares the currently checked out branch against that.
- If no branches are specified, it compares the currently checked out branch against master.
- If a
--filter
argument is specified, it's passed to bench_bitcoin's-filter
. - If
--threshold
or--colour
arguments are specified, they are treated as above. - If one of the branches is checked out already, it builds and benchmarks that one first, to make best use of any cached build artefacts.
- Example usage (basic / advanced):
../contrib/bench/bench_compare_branch.sh
../contrib/bench/bench_compare_branch.sh --filter 'Base58.*' --threshold 15 master my_branch
Example output:
> contrib/bench/bench_compare.sh temp1.bench temp2.bench
Reporting differences greater than 10% threshold.
Total benchmarks: 10, common: 10, unique to bench_gethash_before.txt: 0, unique to bench_gethash_after.txt: 0
CheckBlockHashTest_1MB total time decreased 97.95%, speed increased 4791.38%:
File Benchmark evals iterations total min max median
bench_gethash_before.txt CheckBlockHashTest_1MB 5 1000000 4.98315 9.66528e-07 1.05475e-06 9.72653e-07
bench_gethash_after.txt CheckBlockHashTest_1MB 5 1000000 0.101876 1.95823e-08 2.08268e-08 2.04559e-08
CheckBlockHashTest_32MB total time decreased 97.83%, speed increased 4527.17%:
File Benchmark evals iterations total min max median
bench_gethash_before.txt CheckBlockHashTest_32MB 5 1000000 4.87075 9.6566e-07 9.82757e-07 9.7457e-07
bench_gethash_after.txt CheckBlockHashTest_32MB 5 1000000 0.105264 2.09201e-08 2.13241e-08 2.0992e-08
CheckProofOfWorkTest_1MB total time decreased 94.14%, speed increased 1609.32%:
File Benchmark evals iterations total min max median
bench_gethash_before.txt CheckProofOfWorkTest_1MB 5 1000000 5.26902 1.04402e-06 1.07516e-06 1.04993e-06
bench_gethash_after.txt CheckProofOfWorkTest_1MB 5 1000000 0.308251 6.07236e-08 6.2348e-08 6.17767e-08
CheckProofOfWorkTest_32MB total time decreased 94.05%, speed increased 1580.76%:
File Benchmark evals iterations total min max median
bench_gethash_before.txt CheckProofOfWorkTest_32MB 5 1000000 5.30435 1.03756e-06 1.07703e-06 1.06923e-06
bench_gethash_after.txt CheckProofOfWorkTest_32MB 5 1000000 0.315592 6.20769e-08 6.488e-08 6.30122e-08
The following 4 benchmarks appear only in temp2.bench: CheckBlockTest_1MB CheckBlockTest_32MB CheckProofOfWorkTest_1MB CheckProofOfWorkTest_32MB
Test plan
../contrib/bench/bench_compare_branch.sh master [local_branch_of_your_choice]
Edited by SlowRiot