Project Health Analytics: Design and Validation

1. TL;DR

One theme that emerged in our user research was the need to have an at-a-glance assessment of a project's current health. But what is project health exactly? The following issue frames the problem space and suggests an MVC and research approach to tackle this feature area...

Scope:

• Initial target user: Engineering managers
• Job story: When I am conducting a retrospective, I want to see some quantitative indicators of project health, so that it sparks ideas/discussion for improvement.

Proposal:

1. Ship MVC
2. Conduct broad qualitative research (problem/solution validation mix) using a survey with a few targeted user & SME discussions
3. Conduct quantitative research to inform designs and further areas of validation

2. User context

Target users

1. Gabe, Engineering Director
2. Delaney, Dev team lead
3. Sasha, Software developer
4. Parker, Product Manager

Job-to-be-done

• Sprint/milestone planning
• Retrospectives
• Stand-ups/daily check-ins

Buyer type

Since this is at project level - something like core/starter. But we can incorporate additional dimensions of project health (if necessary) in higher tiers (e.g. security).

3. Research

Competitor/secondary research

• Tasktop - flow metrics
  • Flow Velocity gauges whether value delivery is accelerating. Flow Velocity is the number of Flow Items of each type completed over a particular period of time. This is also referred to as throughput.
  • Flow Distribution helps prioritize specific types of work during specific time frames in order to meet a desired business outcome. Flow Distribution measures the ratio of the four Flow Items completed over a particular window of time.
  • Flow Time can identify when Time to Value is getting longer. Flow Time measures the time it takes for Flow Items to go from ‘work start’ to ‘work complete’, including both active and wait times.
  • Flow Efficiency can identify when waste is increasing or decreasing in your processes. Flow Efficiency is the ratio of active time vs. wait time out of the total Flow Time.
  • Flow Load monitors over and under-utilization of value streams, which can lead to reduced productivity. Flow Load measures the number of Flow Items currently in progress (active or waiting) within a particular value stream.
• Atlassian - Team health monitor workshop - Very interesting because they've approached this problem qualitatively using a workshop exercise. Some of the dimensions they explore:
  • Full-time owner
  • Balanced team
  • Shared understanding
  • Value and metrics
  • Proof of concept
  • One-pager
  • Managed dependencies
  • Velocity
• Spotify - Squad health check model - Again, another workshop-based approach!
• Screenful dashboard - Interesting health stats setup to be shown on a large screen. Show metrics around the following questions:
  • Who's working on what?
  • What are the lead & cycle times?
  • Is the project on time?
  • How is time spent
• shtirlic project health - cmd line plugin - Shows project health from command line and describes it in term of issues and MRs

Existing UX research

From the previous 12.7 research spike, Comments/feedbackfrom PM for a day exercise & interviews:

• "An at a glance view of how things are going seems useful as long as I understand how that "health" is defined and that it is meaningful."
• "In principle the dashboard report is quite valuable for us, but here it would depend on what the specific KPIs are… KPIs vary from project to project"
• "For us to have a meaningful view of large parts (RAG status) of the company would be incredibly useful"
• "Would help PMs create 5:15 project report - 5 mins read, 15 to write… these could be included if you click on the project"
• "Missed milestones, estimation vs actual, velocity. How effective have we been at meeting our milestone plan"

Assumptions/Hypotheses

• Stand-ups, retrospectives & 1:1s currently stand as a more qualitative source of project health data.
• Project health is viewed as more than purely repo health - People (team health) / Process / Product (Vision, direction & quality).
• Project health and team health are different from a GitLab naming convention and IA POV, users will be understand this difference.
• Quant is the what, qual is the why - we can provide value by showing the numbers and providing an avenue for users to investigate/diagnose the qual
• Project health KPIs will vary at different levels of project maturity along an S curve (i.e. Forming, storming, norming performing)

Questions

• What factors have contributed to healthy or unhealthy projects in the past?
• Are there a standard set of project health KPIs that would be useful for most users (80:20)?
• How do project health requirements and KPIs differ when a user is project-minded rather than product-minded?
• Is there a difference between how Engineering Directors would use this compared to Engineering Managers vs ICs?
• Would being remote/co-located impact how this feature is designed?
• What actions would users want to take after seeing the status of their project health?
• In what context would this information be used? What sort of cadence? (e.g. Stand-ups & Retros)
• What's the relationship between alerts/notification and project health?
• Are project health dimensions much different from group or instance-level health?
• Will this be different for monoliths vs micro-service-based vs (other type?) architectures?

4. Solution ideation

Opportunities

• How might we find the right balance between project and team health, and communicate this distinction to our users?
• How might we collaborate with the new teams/spaces feature group to create actionable insights around teams?
• How might we gather qualitative insights in context of quantitative insights?
• How might we facilitate remote teams create/view similar information to that yielded from Spotify/Atlassian workshops?
• How might we provide an abstracted view of multiple project's health?

Design considerations

• Glanceable - Design to be scanned, but allow further investigation/action if necessary
• Actionable
• Abstractable - Project view, multi-project view, group view, multi-group view, instance view
• Tier feature modularity - Must be designed to adapt, strip-away or add data based on the features included in that tier
• Scalable - Include further visualisations in later iterations
• Quant/Qual - Combining quantitative and qualitative data
• Comparable - maybe this could be an interesting feature to explore how we compare multiple projects side-by-side?
• Habit-forming/coaching - visualising and facilitating behaviour change over time

Content - Questions to answer / Potential KPIs

RAG status should be calculated using relative progress improvements over time and ratios rather than absolute/arbitrary values. Infinite metrics rather than finite metrics. ⚙️ = Calculate 🛑🟨🟢 using these metrics

Plan

• Process - Is this project's flow improving? Are there any bottlenecks?
  • Lead & cycle times ⚙️
  • Throughput/velocity ⚙️
• Issues - Are issues being planned and addressed effectively? What's being worked on?
  • Total open
  • Created vs closed ⚙️
  • Completed vs target ⚙️
  • Average resolution time ⚙️
  • Milestone goals & progress
  • Features being delivered
  • Cumulative flow
• Type of work - What is the distribution of the type work items for this milestone/period of time?
  • Bugs
  • Features
  • Security
  • Debt
• Bugs - Are bugs being addressed? Are there any outstanding severe bugs?
  • Total open
  • Created vs closed ⚙️
  • Bugs being addressed in a release (severe bugs are fixed quickly) ⚙️

Create

• Merge Requests - Are MRs being merged effectively?
  • Total open
  • Created vs closed ⚙️
  • Time to merge ⚙️
  • No. of commits
• Repository - How is the code and dev process in the repo?
  • No. of commits
  • Line changes
• Members - Do we have a good balance of active contributors & maintainers?
  • Total
  • Active ⚙️
  • Maintainers / contributors ⚙️
  • Happiness / satisfaction
  • Recent Activity - what's everyone working on?

Verify

• CI/CD - How effectively are we testing and deploying?
  • Test coverage ⚙️
  • Pipelines run ⚙️
  • Successful pipelines
  • Time/pipeline
  • Av. commit duration ⚙️

Ops

• Operations - Are we fixing the right things? Not violating SLAs?
  • Availability ⚙️
  • SLA violations ⚙️

Sec

• Security & Compliance - How many vulnerabilities do we have and what are we doing about them?
  • Total open vulnerabilities
  • Vulnerability distribution: Critical, high, medium, low ⚙️
  • Dependencies

Other

• CSAT / NPS

Layout - variations

Dashboard	One page condensed	Side index nav-scroll	Accordian	Accordian v2

5. Proposal

Quantitative Research

It seems to me that we could get a lot of insight into what makes a healthy/unhealthy project by doing some quantitative analysis across the different broad range of existing customer instances. I imagine these insights could in-turn be used as a bit of a coaching mechanism. Would it be possible to get some data science love on some of the questions in the research section?

Qualitative Research

Because this is a new feature topic with the potential for a lot of different opinions, I'd like to start by casting the net quite wide to get an understanding of what "project health" means and might look like to our users at different levels of hierarchy:

• Survey across ICs, EMs and EDs
• Subject-matter expert discussions
• 1-2 discussions with GitLab EMs
• 1-2 discussions with customer EMs
• Could potentially be bundled with some other research topics

MVC

The first version of this should be at the project level. If this is interesting to our customers, we can consider group and instance-level health indicators.

Success criteria

What does success look like? How can we measure it?

• Design, solution & plan created in %12.8
• Agree on research approach in %12.8
• Build & ship MVC feature within 1 milestone
• Dogfood with team and get GitLab feedback
• Get user/customer feedback on direction