WIP: Compare benchmarks on CI
This MR introduces benchstat
so that we can use CI to prevent performance degradations like gitlab-com/gl-infra/delivery#633 (comment 272847098)
It includes several changes:
- we now run merge_requests pipeline (
GitLab-ci.yml
refactoring) - a new job runs benchstat running the benchmarks 5 times
- the same stat is taken from the merge target and the 2 times are compared
-
fixing our benchmarking as it must loop overextracted into !238 (merged)b.N
(thanks @karrick for noting this)
How to read the output.
On !236 (diffs) I introduced a delay inside the benchmarked function, the benchmark
job on that MR shows
Benchmarks comparison
name old time/op new time/op delta
ReadGroups/10_groups_3_levels-2 5.63ms ± 6% 2051.24ms ± 1% +36362.64% (p=0.008 n=5+5)
ReadGroups/100_groups_3_levels-2 70.7ms ± 2% 2314.6ms ± 0% +3175.30% (p=0.008 n=5+5)
ReadGroups/1000_groups_3_levels-2 3.27s ± 2% 5.36s ± 3% +64.01% (p=0.008 n=5+5)
ReadGroups/10000_groups_1_levels-2 1.56s ± 1% 3.69s ± 2% +136.65% (p=0.008 n=5+5)
Performance degradation detected
make: *** [Makefile.util.mk:27: benchstat] Error 1
Edited by 🤖 GitLab Bot 🤖