Gather user-defined metrics from CI build output via config in yml file
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
Problem to solve
Often times I have tests running in CI that report some quantitative metric. For example, I may have a test report timing info for some critical function. It's difficult to fail the test (and therefore mark a failed CI job) based on this value, and failing may not even be desirable. In order to get this information I need to go to the build output of every job which has the output and do manual analysis.
Further details
Use cases:
- Gather timing information
- Gather code coverage information (in a more generalized way than is accomplished now, including different ways of measuring coverage)
- Gather runtime memory usage
- Enable analysis of how metrics such as those above have changed across the git history and/or across time
- View trends in the metrics over time
- Easily track where a metric dramatically shifts (e.g. one commit doubles the time of a benchmark)
Proposal
Add custom metric specifications to .gitlab-ci.yml. These would be similar to how you can set up code coverage parsing today (in project settings, with a regex) but generalized to be able to parse any user-defined variable.
For each job with metrics specified, variables will be extracted and stored with the job.
Initially, as a minimum-viable-product, this information should be viewable in the pipeline page and also downloadable in a basic CSV format.
In the future, it would be really cool to have metrics plotted automatically to show change in variables with respect to git commits / time.
What does success look like, and how can we measure that?
One would be able to add a custom metric attribute to a .gitlab-ci.yml job, run several builds, and at a glance be able to see the values of the gathered metric across multiple runs and commits.