Allow developers to follow their application's performance
Description
As your piece of software matures, you want to make sure you don't introduce performance regressions into your code, you also want to know if you're making progress when you're doing some performance work.
This idea is inspired from what elastic.co is doing, they have nightly benchmarks such as this one when they can track the overall trends of performance across days/months
In the "From idea to production" mindset, you should be able to automate this task, and not have to setup some custom thing to do a proper benchmarking.
A lot of software could benefit from this, I'm sure gitlab could too :)
Proposal
With this in mind, I'd like to see something like this:
- your software has some performance tests that output to a file when run, with a naming convention like perf-.data (multiple tests can append to the same file, a file is a way to group different metrics into a common graph)
- each file has one value per line. So for example perf-generatestuff.data has 4 lines: avg, max, min, std deviation. Those values are the value in the y axis. The x axis value is taken from the commit time.
- in your .gitlab-ci.yml, you specify the list of so it knows which data files to pick up (optional).
- for each build, the CI system picks up the data files, add them to a table.
- you can then go to a performance tab for your project and see all the graphs. On that page you have as many graph as different files you generate for each build
- Each graph is build by reading the table and plotting the corresponding graphs.
Then it becomes easy and natural for developers to track key performance metrics for their app ! Gitlab's world domination: done ;-)
Links / references
The two top links when you type "benchmarking application nightly" in google: