Create new UMAU metric (UMLU) for users with a last_activity_on date within the previous 28 days
Problem to be solved
Today GitLab relies on UMAU to understand how many users are active in the self-managed instance or SaaS namespace. This is then used to determine renewal and expansion. However our UMAU today is only capturing a small set of activities and is therefore not the right measure. We need the most accurate representation of UMAU so that both our customers and everyone at GitLab has accurate information to make business decisions.
Background
GitLab defines Unique Monthly Active Users (UMAU) as:
The number of unique users that performed an event within the previous 28 days for Self-Managed and SaaS.
An event
is a narrowly-defined set of actions performed against a set of objects within GitLab.
Separately, as part of the response body from API resources including Users and Members, GitLab provides a last_activity_on
attribute, which is set to the current ISO-formatted date (ex. 2021-06-15
) each time any of a broader set of actions is observed from a user.
Proposal
Create a new metric, UMLU - Unique Monthly Logged in Users
- These are users that have a last_activity_on date within the past 28 days.
- Noting that last_activity_on encompasses a broader set of actions and is a more inclusive measure of active users (picks up active users even when they are not using the UI).
- This metric indicates any level of activity.
Justification
It is typical for customers to leverage the last_activity_on
field when identifying "idle" user accounts, and [optionally] block or deactivate them. This is because last_activity_on
encompasses a broader set of actions and is a more inclusive measure of active accounts.
Beyond the proportional indications that UMAU can offer GitLab of user growth and adoption, UMAU is valuable as a predictor of retention; i.e. accounts approaching renewal with a UMAU significantly below their number licensed seats present a greater risk of (partial) churn. It is only accurate if it reflects how organizations actually react to user activity.
Supporting Evidence: A large Enterprise Self-Managed Customer with 7501
licensed seats was recently showing a UMAU of ~3800
, but a unique count of ~5400
users with a last_activity_on date within the preceding 30 days. At a Premium list price, the difference of 1600 users between the two counts represents a significant delta (>$350k) in ARR risk.
Details
last_activity_on
is updated by the following user actions (per https://docs.gitlab.com/ee/api/users.html#get-user-activities-admin-only)
- Git HTTP/SSH activities (such as clone, push)
- User logging in to GitLab
- User visiting pages related to Dashboards, Projects, Issues, and Merge Requests (introduced in GitLab 11.8)
- User using the API
- User using the GraphQL API
Whereas Events
is a richer, but narrower reflection of user activity, (reference https://docs.gitlab.com/ee/api/events.html#action-types) defined by a variety of actions
:
- approved
- created
- updated
- closed
- reopened
- pushed
- commented
- merged
- joined
- left
- destroyed
- expired
performed against an object
:
- issue
- milestone
- merge_request
- note
- project
- snippet
- user
Our current equivocation of UMAU to the unique count of authors of Manage=>Events objects within Usage Ping is missing (at minimum):
- Users who just log in but do not create an action (eg. PMs, senior leaders, etc)
- Users who only clone repos (perhaps SREs)
-
??? validation needed ???
Users who push/commit but not in the context of an MR (i.e. if i just push to a repo, and dont use a GitLab MR, does that create an Event? - Users who only use the API to read data (e.g. service users)