Skip to content

GitLab Next

GitLab Data Team
GitLab Data Team
Closed
Opened by Mark Wood@mjwood🌀Developer4 of 10 tasks completed4/10 tasks

Dev:Plan:Certify NSM

⭐️ North Star Metric - WIP

📖 Background and Overview

▶ Define your product feature's key benefit/value to customers

Requirements Management offers customers a way to create a definition of their product features, and provides a single source of truth for correct functionality of a product.

In the context of a regulated industry, all software must be in relation to the implementation of a requirement. In addition, all testing must be verifying a requirement.

🥅 Define north star metric that quantifies delivery of this "must have" value

As Requirements Management becomes more mature, we want to transition from measuring success based on number of requirements created to interactions with requirements.

A general workflow for teams utilizing requirements would be as follows:

  1. Create high level requirements (requirements at a group level) - not yet implemented in GitLab
  2. Create low level requirements (requirements at a project level) - Implemented in GitLab %"12.10"
  3. Develop design / code / testing by referencing requirements
  4. Link appropriate design / code / test to requirements - Test linking MVC in %"13.1"

In short, the actual creation of requirements is only a small piece of capturing the benefits of having a requirements based development approach - the far greater measure of success is to measure the interactions with requirements - specifically viewing and tracing to requirements.

Proposal

At present, we are able to see requirement creation metrics for self-managed instances via Usage Ping. We would like to plan out and implement the next steps as follows:

  1. Track requirement navigation changes - are users paging through requirements?
  2. Track requirements filter events - are users searching for requirements to reference them?
  3. Track linking events - are users linking both code and tests to requirements?
  • Once defined, please review with this checklist and iterate if needed
  • Does this support sustainable revenue growth?
  • Does this reflect customer getting value?
  • Does this measure progress?
  • Why do you pick this north star metric for your feature/group/stage?

The single biggest revenue driver for Plan:Certify has shown to be the development of integrated Requirements Management.

AMAU Metric

  • What is the related AMAU metric calculation?

❗ What do you need from the Telemetry/Data team?

  • The data is available but I need analytics to visualize/re-structure
  • The data is not available and I need to implement tracking
  • I don't know, need help to check whether this data exists or not

  • Please provide more details

  • What is the data source for the proposed north star metric? (Link, Explanation of how this data is captured, etc)

  • Who would be the best contact for the data source system? (Link or explanation)

🧩Answer these questions to provide more context

  • Any alternative north star metrics you would consider, either because they are easier to get, or you are debating between that and the proposed NSM?

We have some metrics surrounding Service Desk usage, but with Service Desk moving to core, it is not longer as significant value driver as Requirements Management.

  • Does your feature play a bigger role in initial purchase, retention, or expansion/upgrade?

Based on discussions with customers, initial purchase and expansion/upgrade are both likely outcomes.

  • Does your feature play a bigger role in SMB sales, Enterprise sales, or both?

Enterprise sales

  • What are some dimensions you would expect to slice/dice your north star metric? For example, by plan tier, by free/paid, etc.

Requirements Management is only available in Ultimate. I would like to understand what size enterprise is utilizing requirements.

  • Are there any GitLab product features that are "downstream" or "upstream" to your product feature? For example, customers have to use feature A to use your feature, or once customers use your feature, they are more likely to use another feature.

Given that we will be able to satisfy requirements through testing, this will hopefully drive additional usage of pipelines.

  • Please attach a mock-up, drawing, or chart in another system that will think will allow you to make a data-driven decision for the propose metric. Link: ______
    • Please provide an example of how you would use the mock-up or chart to make the data-driven decision.

📈 Define Supporting KPI/metrics - [Optional]

  • Once defined, please review with this checklist and iterate if needed
  • Are these metrics an input into north star metric?
  • By improving these, do you improve the north star metric?