Create new, outcome-based engineering productivity metric
Proposal for first version:
Satisfaction and well being of our team (Diversity, Belonging, and Inclusion)
- Quarterly Engineering satisfaction survey NPS score
- Team member retention (example)
Performance (Results, Efficiency)
- Escape Rate - Tableau
- Flaky tests - Tableau
- Change failure rate (CFR)
- Mean time to recover (MTTR)
- Number S1 bugs
Activity (Results, Efficiency)
Communication and collaboration (Collaboration, Transparency)
- Time from hire to first contribution (new metric)
- Number of teams involved in an MR (new metric)
Efficiency and flow (Efficiency)
- Review Time to Merge - Tableau
- Open MR Age - Tableau
- Open Bug Age (OBA)
- Deployment frequency
- Lead time to Change
Why
Currently our KPIs are:
And our PIs are:
There are some factors here that I'd like to resolve:
- These [K]PIs don't speak to GitLab customer outcomes or internal customer outcomes
- These [K]PIs don't encapsulate the work the team is responsible for [1, 2, 3]
Proposal
Develop a KPI based on the SPACE model that encapsulates multiple factors to give a broad overview of the productivity of our team.
This KPI should be:
- an outcome
- something we can directly contribute to
- customer results focussed
- proximate enough that the value is clear
Summary of SPACE and values alignment:
- Satisfaction and well being of our team (Diversity, Belonging, and Inclusion)
- Performance (Results, Efficiency)
- Activity (Results, Efficiency)
- Communication and collaboration (Collaboration)
- Efficiency and flow (Efficiency)
Independent aspects of this should have their own metrics created where they don't already exist.
Considerations
- Metrics related to all of engineering will need input from other teams within GitLab - communication is important.
- One of the aspects of a metric like this is surveys, this will require collaboration from other teams.
- Metrics for specific areas of the SPACE model of productivity may be new for teams (where they don't already exist) and may not be able to be adopted.
- A change to how Engineering Productivity is measured can't be made in isolation.
- Engineering Productivity should account for all engineers at GitLab, not just folks working in the Development Department.
- Change takes time, iteration is one of our values.
What's next
- Break down each aspect of SPACE
- Identify appropriate metric, where missing, develop one
- Build model to integrate these metrics
- Build and test SPACE metric
Timeframe
Ideally we have something to work with before the end of the quarter so that we can move into FY24Q2 with a clear vision and method for measuring our progress.
Feedback / Questions
I'll add a thread below for feedback and questions!