Define key test tooling metrics for Test Platform
Why
If you can’t measure it, you can’t improve it. – Peter Drucker
At Gitlab, we care about Results and adhere to the principles of "we can’t assume that what works well for us will support our customers’ needs. When we have an idea, we must directly validate our assumptions with customers to ensure we create scalable, highly relevant solutions.". The testing solutions should improve the right measurable outcomes. We need to identify what we should be measuring to improve the overall test tooling that can help shipping better quality products faster.
Finally these specific segments in the Gitlab OKR handbook summaries the need for identifying the key metrics -
- "OKRs stand for Objectives and Key Results and are our quarterly objectives. OKRs are how to achieve the goal of the Key Performance Indicators KPIs. "
- "As Key Results provide the milestones for how we’ll complete objective, KR should be either a qualitative (i.e. completed Y/N or number of steps of project completed) or quantitative (increased a metric by x) measure that can prove we accomplished the Key Result. Quantifying Key Results strongly preferred."
Proposal:
Click to expand
- Test Effectiveness Metrics:
- Definition: Metrics that measure the thoroughness and relevance of testing activities. The Test & Tools infrastructure improvements finally be contributing to the effectiveness. Without the effectiveness metrics, the existence of the team doesn't make sense.
- Intent: To ensure that tests cover all critical aspects of the software, leading to robust and reliable code.
- Why Evaluate: Effective testing is crucial for identifying potential issues early, reducing the risk of bugs in production, and ensuring a high-quality product.
- Metrics:
- Test Coverage: Focus on promoting uniform test coverage across static, unit, integration, and end-to-end tests. The key factor is how TTI team is contributing to the coverage improvement & how it is removing the road blockers to improve coverage.
- Defect Rate: Track defect/bug rate over a period of time. For TTI team, The key factor tracking bugs through automated testing/tooling.
- Test Efficiency Metrics:
- Definition: Metrics focusing on the speed and resource utilization of testing processes.
- Intent: To streamline testing procedures, making them faster and more cost-effective.
- Why Evaluate: Efficient testing processes save time and resources, allowing for more agile development cycles and quicker responses to market or technology changes.
- Metrics:
- Test Execution Time: Monitor the duration of test executions to optimize performance.
- Test Reliability Quotient: Measure and reduce the inconsistency in test flakiness.
- Blocking Deployments: Monitor instances where bugs or failing tests delay deployments.
- Master Branch Stability: Collaborate with the productivity team to ensure stable mainline development.
- Test Experience Metrics:
- Definition: Metrics that assess the experience of both developers and users in relation to testing.
- Intent: To ensure that testing tools and processes are user-friendly and effectively meet the needs of developers and end-users.
- Why Evaluate: A positive testing experience encourages more frequent and thorough testing, leading to better overall software quality and user satisfaction.
- Metrics:
- Time to User Activation: Evaluate the speed of bringing users into the testing process.
- Time to Support Resolution: Measure the responsiveness of the support requests.
- Developer Contributions: Track the involvement of developers in testing contributions.
- Developer Feedback and Satisfaction: Gather and analyze feedback to assess developer contentment with testing tools and processes.
- Test Economy Metrics:
- Definition: Metrics assessing the financial aspects of the testing process.
- Intent: To ensure cost-effectiveness in testing, balancing quality with resource expenditure.
- Why Evaluate: Understanding and managing the costs associated with testing is vital for maintaining budgetary constraints and ensuring that resources are allocated effectively.
- Metrics:
- Test Execution Cost: Assess the financial efficiency of test executions.
- Team Operational Cost: Monitor the overall expenditure of the TTI team.
- Team growth & satisfaction metrics
- Metrics:
- Measure overall satisfaction and engagement through surveys to gauge team morale and identify areas for improvement.
- Monitor turnover & retention rates to understand how well the team environment supports long-term employee retention
- Track the number of professional development activities engaged in by team members, indicating commitment to growth and learning.
- Metrics:
Note:
- This metrics will also be important for Self-managed platform team tooling, the TTI team will only concentrate on the tooling that TTI owns. Some of these can be top level metrics for Test-Platform also.
- We will always define metrics based on the responsibilities given to the TTI team -
- Best in class Test distribution Strategy
- Test framework enhancements
- GitLab QA
- CustomersDot
- Improve pipeline efficiency
- Reduce flakiness
- Faster execution
- Increase test reliability
- Test Environment enhancements
- Efficiency tools and AI initiatives
- Product Analytics tooling (product section)
- Pipeline DRI on-call improvements
- Support Project Runway
- Test environment - Staging-ref
Expected Outcome
- We know what to measure for TTI team to be successful
- We know how to measure the metrics
- If any data is available, integrate with the TTI handbook page
Action Plan
Process Improvement
-
Simplify process of data warehousing - &71
Defect Triaging
-
TBD
Test Reporting
Test Tooling
-
Develop dashboard for Test Execution Time - #2244 (closed) [Dashboard] -
Develop dashboard for Test Economy Metrics - https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/work_items/2245 [Dashboard] -
Develop dashboard for Test Flakiness Metrics - #2259 (closed) [Dashboard] -
Develop dashboard for Test Infra Availability Metrics - #2260 (closed) [Dashboard] -
Establish Alerts and SLOs for the Testing Tooling Dashboard Monitoring - #2591 -
Extend test tooling dashboard for non-E2E tests - &68
Developer Contribution
Productivity
-
Develop On-call SET metrics - #2543 (comment 1827633260) -
Develop internal dashboard for Team Productivity Metrics - #2463 (pilot with TTI) [Dashboard WIP] -
Test platform growth and satisfaction metrics - #2184 (comment 1791670864)
Reference
We can also take inspiration from how Engineering Productivity is keeping their KPIs easily measurable and justify the business impact - https://about.gitlab.com/handbook/engineering/quality/engineering-productivity/#kpis