Spike Research: Using ClickHouse as the datastore for the Exec Dashboard
Problem to solve
Data-driven optimization of SDLC requires a centralized view of all the DevOps metrics that available in GitLab.
- Users need to aggregate multiple data records from multiple stages into one view.
- PostgreSQL is not set up for analytical workloads.
- Analytics data is currently spread between multiple different APIs
Suggested solution
- Daily aggregation jobs to extract the metrics and copying them to ClickHouse.
- The granularity of the measurement will be on daily intervals.
- The Dashboard will query the data from ClickHouse. (Out of scope for this issue)
Reference use cases
Metrics list
- Deployment Frequency (DORA)
- Lead Time for Changes (DORA)
- Change Failure Rate (DORA)
- Time to Restore Service (DORA
- Lead time (VSA)
- Cycle time (VSA)
- MR rate
- Commits rate
- Critical Vulnerabilities
- Project Security status
- DevOps Score
- DevOps Adoption by subgroup
Use case A: Group by: Project
Click to expand
Use case B: Group by: Metric
Click to expand
User experience goal (Out of scope for this issue)
- #370887 (closed)+
- Data retention policy - 3 years
- Aggravated view of 200 groups, around 8 projects per group - total of 1600 projects.
Investigation and clarification questions:
- What about self-managed ?
- What about drill down from the Dashboard into the other analytic pages?
Related information
Edited by Haim Snir