Gather user-defined metrics from CI build output via config in yml file
Problem to solve
Often times I have tests running in CI that report some quantitative metric. For example, I may have a test report timing info for some critical function. It's difficult to fail the test (and therefore mark a failed CI job) based on this value, and failing may not even be desirable. In order to get this information I need to go to the build output of every job which has the output and do manual analysis.
Further details
Use cases:
- Gather timing information
- Gather code coverage information (in a more generalized way than is accomplished now, including different ways of measuring coverage)
- Gather runtime memory usage
- Enable analysis of how metrics such as those above have changed across the git history and/or across time
- View trends in the metrics over time
- Easily track where a metric dramatically shifts (e.g. one commit doubles the time of a benchmark)
Proposal
Add custom metric specifications to .gitlab-ci.yml. These would be similar to how you can set up code coverage parsing today (in project settings, with a regex) but generalized to be able to parse any user-defined variable.
For each job with metrics specified, variables will be extracted and stored with the job.
Initially, as a minimum-viable-product, this information should be viewable in the pipeline page and also downloadable in a basic CSV format.
In the future, it would be really cool to have metrics plotted automatically to show change in variables with respect to git commits / time.
What does success look like, and how can we measure that?
One would be able to add a custom metric attribute to a .gitlab-ci.yml job, run several builds, and at a glance be able to see the values of the gathered metric across multiple runs and commits.