Skip to content

Contrib script to compare benchmark results

SlowRiot requested to merge SlowRiot/bitcoin-cash-node:bench_compare into master

This MR adds two new scripts:

  • contrib/bench/bench_compare.sh: Compare results of two benchmark runs, and report the biggest changes in both time elapsed and speed of execution, by percent, sorted by the change amount.
    • By default, reports only changes greater than 10%. This is configurable with the --threshold option.
    • By default, reports only changes greater than 10%. This is configurable with the --threshold x option.
    • Colour output is enabled by default in interactive terminals, disabled otherwise, controlled by both --colour and --color flags to allow for transatlantic flexibility ;) Accepts "on", "off" (and common synonyms like "true", "false"), and "auto".
  • contrib/bench/bench_compare_branch.sh: Check out, build and run benchmarks on two branches, and compare their results.
    • If one branch is specified, it compares the currently checked out branch against that.
    • If no branches are specified, it compares the currently checked out branch against master.
    • If a --filter argument is specified, it's passed to bench_bitcoin's -filter.
    • If --threshold or --colour arguments are specified, they are treated as above.
    • If one of the branches is checked out already, it builds and benchmarks that one first, to make best use of any cached build artefacts.
    • Example usage (basic / advanced):
      • ../contrib/bench/bench_compare_branch.sh
      • ../contrib/bench/bench_compare_branch.sh --filter 'Base58.*' --threshold 15 master my_branch

Example output:

> contrib/bench/bench_compare.sh temp1.bench temp2.bench
Reporting differences greater than 10% threshold.
Total benchmarks: 10, common: 10, unique to bench_gethash_before.txt: 0, unique to bench_gethash_after.txt: 0

CheckBlockHashTest_1MB total time decreased 97.95%, speed increased 4791.38%:
    File                      Benchmark               evals  iterations  total     min          max          median
    bench_gethash_before.txt  CheckBlockHashTest_1MB  5      1000000     4.98315   9.66528e-07  1.05475e-06  9.72653e-07
    bench_gethash_after.txt   CheckBlockHashTest_1MB  5      1000000     0.101876  1.95823e-08  2.08268e-08  2.04559e-08

CheckBlockHashTest_32MB total time decreased 97.83%, speed increased 4527.17%:
    File                      Benchmark                evals  iterations  total     min          max          median
    bench_gethash_before.txt  CheckBlockHashTest_32MB  5      1000000     4.87075   9.6566e-07   9.82757e-07  9.7457e-07
    bench_gethash_after.txt   CheckBlockHashTest_32MB  5      1000000     0.105264  2.09201e-08  2.13241e-08  2.0992e-08

CheckProofOfWorkTest_1MB total time decreased 94.14%, speed increased 1609.32%:
    File                      Benchmark                 evals  iterations  total     min          max          median
    bench_gethash_before.txt  CheckProofOfWorkTest_1MB  5      1000000     5.26902   1.04402e-06  1.07516e-06  1.04993e-06
    bench_gethash_after.txt   CheckProofOfWorkTest_1MB  5      1000000     0.308251  6.07236e-08  6.2348e-08   6.17767e-08

CheckProofOfWorkTest_32MB total time decreased 94.05%, speed increased 1580.76%:
    File                      Benchmark                  evals  iterations  total     min          max          median
    bench_gethash_before.txt  CheckProofOfWorkTest_32MB  5      1000000     5.30435   1.03756e-06  1.07703e-06  1.06923e-06
    bench_gethash_after.txt   CheckProofOfWorkTest_32MB  5      1000000     0.315592  6.20769e-08  6.488e-08    6.30122e-08


The following 4 benchmarks appear only in temp2.bench: CheckBlockTest_1MB CheckBlockTest_32MB CheckProofOfWorkTest_1MB CheckProofOfWorkTest_32MB

Test plan

  ../contrib/bench/bench_compare_branch.sh master [local_branch_of_your_choice]
Edited by SlowRiot

Merge request reports