Skip to content

WIP: Compare benchmarks on CI

Alessio Caiazza requested to merge benchstat into master

This MR introduces benchstat so that we can use CI to prevent performance degradations like gitlab-com/gl-infra/delivery#633 (comment 272847098)

It includes several changes:

  • we now run merge_requests pipeline (GitLab-ci.yml refactoring)
  • a new job runs benchstat running the benchmarks 5 times
  • the same stat is taken from the merge target and the 2 times are compared
  • fixing our benchmarking as it must loop over b.N (thanks @karrick for noting this) extracted into !238 (merged)

How to read the output.

On !236 (diffs) I introduced a delay inside the benchmarked function, the benchmark job on that MR shows

Benchmarks comparison
 name                                old time/op  new time/op     delta
 ReadGroups/10_groups_3_levels-2     5.63ms ± 6%  2051.24ms ± 1%  +36362.64%  (p=0.008 n=5+5)
 ReadGroups/100_groups_3_levels-2    70.7ms ± 2%   2314.6ms ± 0%   +3175.30%  (p=0.008 n=5+5)
 ReadGroups/1000_groups_3_levels-2    3.27s ± 2%      5.36s ± 3%     +64.01%  (p=0.008 n=5+5)
 ReadGroups/10000_groups_1_levels-2   1.56s ± 1%      3.69s ± 2%    +136.65%  (p=0.008 n=5+5)
 Performance degradation detected
 make: *** [Makefile.util.mk:27: benchstat] Error 1
Edited by 🤖 GitLab Bot 🤖

Merge request reports