CI/CD reporting and analytics

Problem to solve

Users implementing or optimizing their CI/CD pipelines need data to make decisions or to understand areas for improvement.

Further details

For example:

• Which tests take the most time?
• Which tests are flaky (fail often unrelated to code changes)?
• How has the pipeline duration changed over time? Are we getting better or worse? When actively trying to improve pipeline duration, can we track progress?
• Would splitting our pipeline into more parallel jobs be worth it?
• What is the right number of parallel jobs to optimize wall-clock time while paying attention to cost?
• How much does our CI/CD pipeline cost us?
• How much time is spent queueing (waiting for runners to be available)?
• How is our pipeline?
• How often does master break? Who/what most often causes it?

Proposal

• Either take over existing "Charts" page (and rename it) or introduce a new page for reports and analytics
• Look at common usage on Jenkins for target customers to see what they want most, understand what they need most

What does success look like, and how can we measure that?

1. Teams are able to understand their CI/CD pipeline, make improvements, and measure improvements
2. Teams are able to identify areas most deserving of attention

Links / references