CI/CD Metrics
Description
One of the major features I miss when using GitLab CI versus some of the competing products such as Jenkins is the ability to have graphs that show metrics from the CI builds over time, including but not limited to test results (passes and failures), CLOC statistics and the results from static analysis tools. Being able to see these in a timeline across builds is extremely powerful for tracking the progress of a project and ensuring that regression is kept under control. This proposal suggests a method of communicating the results from build plugins with the build co-ordinator and allowing the user to create custom metric dashboards that show these metrics over time for their project.
Proposal
The basics of the proposal is that some format is created that metrics can be output to from build plugins and tools that can then be uploaded back to the build co-ordinator similar to an artifact so that contains information about the metrics that resulted from the build. I propose that the format created be parsed in YAML (which is fully compliant with JSON) allowing either format to be used. If this proposal is feasible then I will be happy to provide more detail on the proposed structure of this metrics file.
With the project, some .gitlab-metrics.yml
will define a list of metrics that can be collected from the builds and specify how these metrics should be treated (what type of graph to show them with, what units to use, which metrics to combine onto the same graph, etc.). This will basically be a declarative document specifying how the metrics UI should look and what the metric names are to populate these graphs with. Keeping this as part of the version controlled repository will have the same advantages as with keeping the .gitlab-ci.yml
part of the version controlled allowing metrics of a specific commit to be visualised. Additionally, this file can specify thresholds which can then be used to mark a build as "unstable" or email the developers a warning. Again, should the proposal be deemed feasible, I can propose a specific structure for this file.
In builds, the metrics file can be specified with a metrics
key for each job in the .gitlab-ci.yml
for which the full format would be something like:
job_name:
metrics:
- file: "target/metrics.yml"
include:
- metric_name_1
exclude:
- metric_name_2
The include and exclude will allow either only specific metric patterns to be included or exclude a pattern of metric IDs. The only required field is the file name and so it may be possible to support shortened formats such as:
job_name:
metrics: "target/metrics.yml"
job_name:
metrics:
- "target/metrics.yml"
Collecting these metrics will be down to build tools created for the specific language and build system being used. For example, a Maven plugin can be created that can analyse the outputs of common plugins such as JUnit, Cobertura, JaCoCo, Checkstyle, FindBugs, Copy and Paste Detector and Programming Mistake Detector to create an output metrics file.
By default the metrics tab that would be created in the UI will display for the default branch, but it will be possible to view this tab for any branch or commit. The view will be constructed based on the .gitlab-metrics.yml
at the HEAD
of the branch, and show metrics collected from all builds in its causal history. This will be particularly convenient separating the metrics from feature branches allowing them to be seen but not to affect the metrics for the mainline until they are merged in. It may be preferable in some cases to only show the build metrics from build which were built with the same tag and may be something to consider.
The metrics will be an aggregation of the metrics collected from all jobs in the build pipeline (which is where the include
and exclude
at the job level will be useful) with later stages in the build process taking precedence should there be two builds providing the same metric. Having this will allow different build jobs to collect different metrics which can be useful.
Please let me know your thoughts on this proposal.