Add a 'buy ci minutes' option in the top right drop down (user settings menu) on GitLab.com
Artifacts
UX Scorecard walkthrough - YouTube video
UX Scorecard Recommendations Epic
Experiment summary
We believe that if we display a buy more ci minutes
button in the top-right user's drop-down menu when a user has less than 30% of ci minutes remaining we will increase add on minutes purchased.
To verify that, we will display a buy more ci minutes
button for the user to engage with which will direct them to the pipeline quota tab in the user's settings of GitLab.com. From here the user can see the minutes they've used what is remaining and be presented with an option to buy. We'll create a funnel report to measure the click-through rate of the new button and also count the subsequent clicks of the buy more ci minutes button on this page.
As a secondary measure, we'll watch the impact on ci minutes purchased here.
Hypothesis
If we display a buy ci minutes button in the user setting drop-down, we'll increase ci minutes purchased.
Personas we're focusing on
Business problem
JTBD: When I realize that we're running out of CI minutes, I want to quickly and easily buy more, so that our team can continue building and delivering software uninterrupted.
Supporting data
In addition to what we already know about ci minutes
We'll want to gather the following data before we start if possible:
- How many users fall into the usage band of
30% ci minutes remaining
- How many users fall into the usage band of
5% ci minutes remaining
Expected outcome
Easy access to the billing page that will present the user with options to add ci minutes to their account. This button will lead them on to the pipeline quota page on GitLab.com
Experiment design & implementation
For this experiment, we will be displaying different button variants to drive awareness around CI Minutes. If possible we'll want to leverage the reaming minutes logic used today when displaying banners.
- The first button will be a basic button that will be displayed to all users who have pipelines set up on GitLab.com
- The second button is a different version of the 1st button but we'll add some additional copy to the button that calls out what % of ci minutes they have remaining. We created two versions of this a
30%
and a5%
so the user sees it change as they get closer to running out of CI minutes.
Designs
Version | Variant | Designs |
---|---|---|
A | Buy CI Minutes |
Link |
B | Buy CI minutes |
Link - short name & Link - long name |
- Runtime: 30 days or until we reach statistical significance
- We will roll this out behind a feature flag and expose this to 20% of users to start then ramp it up from there.
ICE score
Impact | Confidence | Ease | Score |
---|---|---|---|
3 | 7 | 8 | 6 |
Known assumptions
- We'll only show this button to users who have pipelines
- We'll exclude users who do not have pipelines
- We'll limit who we display this to based on the threshold we set (30% or 5% of minutes remaining)
Tracking and reporting
We will need to add tracking to report on this experiment, let's identify our approach to this during the workflowplanning breakdown and include either @eli_kastelein or @mpeychet_
Tracking
-
We will need to track the total # of times we present this button to our users -
We will need to track the total # of time the buy more ci minutes
button is clicked in the drop down -
We will need to track the total # of times the buy additional minutes
button is clicked from the pipeline quota page
Reporting
-
create a report that summarizes the counts for this experiment -
We will need to create a summary report of the revenue impacts as well.
Results, lessons learned, next steps
Follow on experiments (what we're not doing in this experiment)
- Add a notification dot when 'buy ci minutes' button is displayed
- Deep link to the ci minutes purchase screen in the customers app from the user drop down
Checklist
-
Fill in the experiment summary and write more about the details of the experiment in the rest of the issue description. Some of these may be filled in through time (the "Result, learnings, next steps" section for example) but at least the experiment summary should be filled in right from the start. -
Add the label of the Growth subgroup that will work on this experiment. -
Mention the Product Manager, Engineering Manager, and at least one Product Designer from the group that owns the part of the product that the experiment will affect. -
Fill in the values in the ICE score table ping other team members for the values you aren’t confident about (i.e. engineering should almost always fill out the ease section). Add the ICE Score Needed label to indicate that the score is incomplete. -
Replace the ICE Score Needed with an ICE low/medium/high score label once all values in the ICE table have been added. -
Mention the [at]gitlab-core-team team and ask for their feedback.
workflowplanning breakdown
Open items to track down before we move to-
Find the existing logic used when displaying x% remaining minutes banner. -
do a final review of the copy we are using in the designs