Implement alerting on performance regressions

We should have alerting ready in case one of the benchmarks we have wired up via continuous benchmarking regress significantly.