Push the patch release pressure info to the delivery-metrics
Context
Release tools has a dedicated pushgateway to store and keep in memory the delivery metrics (details on &512). As part of Improve patch release process in preparation fo... (&828 - closed) we need to push the patch release pressure info to the pushgateway so it can be used later to adapt the release manager dashboard #2679 (closed).
Proposal
Based on the dashboard updates proposed on Update the release manager dashboard to account... (#2679 - closed), the metrics for the patch release should be able to calculate:
- The total number of merge requests merged into stable branches
- The total number of S1/S2 merge requests merged into stable branches
- A breakdown of S1/S2 merge requests merged per version.
- A breakdown of merge requests merged per version regardless of severity.
We need to implement a new ruby class that pushes the new pressure to delivery-metrics.
Implementation details
The below details were discussed by delivery team members. Link to the call can be found here
General info
- The patch release-pressure was built on #2680 (closed). It's okay to adjust it as long as the new blog post implementation continues to work
- The pushgateway can be found at the
metrics
directory on release-tools - Examples of ruby code pushing metrics to delivery-metrics can be found under the
lib/release_tools/metrics/
directory. - It's important to notice there are two ways of recording metrics:
- The old one (to be deprecated) - Uses the
Metrics::Push.new(<METRIC>)
. An example can be seen onReleasePressure
- The new one - Uses
Metrics::Client.new
. An example can be seen onauto_deploy_pressure.rb
. This is the one that we need to use.
- The old one (to be deprecated) - Uses the
- Along with a Ruby class, the
metrics.go
class should be updated to:- Register the new metrics (
initAutoDeployMetrics
is a good example of this) - Define the labels (versions and severity). The versions should consider three versions back and around 10 versions in the future
- Initialize the metrics by expanding
InitMetrics
on it - An acceptance test is required, it should be added into
TestAPISet
onacceptange_test.go
- Register the new metrics (
Delivery-metrics
- The current release pressure metric (
delivery_release_pressure
) can be reused. Prometheus allows distinguishing between jobs that were pushed byrelease-pressure
(old system) anddelivery-metrics
(the new system) - When a metric is defined, the labels and the possible values should also be defined (this is by design). Since versions (
15.5
,15.6
,15.7
) are dynamically generated this imposes a problem. Two possible solutions for this:Short-term (preferred): Push hardcoded versions (up until 16.0 or 17.0) and then create a follow up to make them dynamic.- Long-term: Declaring a regex that validates versions on the go. (A ready made validator was implemented in gitlab-org/release-tools!2070 (merged) it can be applied to the new metrics we need to define)
Deploying the metrics
- Delivery metrics are continually pushed to the pushgateway
- The metrics pushed are defined in the
metrics
rake task - A pipeline schedule in ops pushes these metrics every hour to the pushgateway (example)
Testing
- Metric can be verified in Thanos - example
To do
-
Add a ruby class and a go function that push the patch pressure information to delivery-metrics - gitlab-org/release-tools!2091 (merged) -
Test the metrics are visible in Thanos #2690 (comment 1215243600)
Edited by Mayra Cabrera