Capture and display types of deployment blockers on grafana
Context
We currently have deployment failures automatically captured under release tasks issues. Release managers get assigned to label them with appropriate RootCause::*
labels, so that by the start of the following week Monday, the deployments:blockers_report
scheduled pipeline under release/tools repo looks through the labelled issues, and makes a weekly deployment blockers issue, like this one.
We want to capture the trend of re-occuring root causes of the deployment blockers, so we can capture and display the changes in frequency as we make modifications to our processes. Recently, RootCause::Flakey-Tests
have been very frequent, and we want to keep track of the trend while we work alongside grouptesting to decrease this count.
This issue is to emit metrics to grafana that captures the types of root causes, and use that metric to create a dashboard panel with trends (weekly or monthly).
Proposal
Rough ideas, feel free to iterate based on feedback and as you see fit.
- In release_tools repo, create a class
ReleaseTools::Deployments::BlockersMetrics
to emit adelivery_deployment_blocker_count
anddelivery_deployment_hours_blocked
metrics that contains the following information in a metric- Label
root_cause
- Label
week
- Label
target_env
- Label
- Implement a rake task in deployments.rake to create + execute that class
- Call that rake task in automation.gitlab-ci.yml, in
record-deployment-blockers
, afterblockers_report
andblockers_annotate
rake tasks - After the MR is merged, run
deployments:blockers_report
scheduled pipeline in ops release/tools. The metric should now appear on grafana (explore) - Use that metric to create a new dashboard panel that shows the trend of blockers (IaC under runbooks/dashboard)
- Note: PoC using the playground first, iterate on feedback, before IaC
Exit Criteria
-
Metrics to capture the various information about deployment blockers are emitted to Grafana -
A dashboard exists on Grafana that shows the trend of deployment blockers per root cause. -
Updated Deployment Blockers Report template that includes the Grafana dashboard link.