Skip to content

For free users of GitLab.com test the demand for trials of bronze and silver

Experiment summary

We believe that by providing users the option to trial Bronze and Silver along with Gold we will increase the total IACV we generate from trials.

To verify that, we will run an experiment where a cohort of free users has the option to trial Bronze, Silver and Gold and compare the IACV to the control where users can only trial Gold.

Hypothesis

We believe that we can increase the amount of IACV generated from trials by offer trials of all tiers (Bronze, Silver, Gold).

Business problem

Currently, users can only start a Gold trial, we may be losing out on trial demand from users that are only interested in Bronze or Silver.

Supporting data

Today users can only trial Gold, in periscope we found that of all the trial upgrades in the last 3 months, 7.07% upgrade during or after the trial. Of that 7.0% that upgraded, 6.22% purchased Gold, 39.74% purchased Silver and 54.03% purchased Bronze (source).

Expected outcome

We expect that by increasing the overall volume of trials by allowing users to select trials of Bronze and Silver we will increase the total IACV generated compared to only allowing users to pick gold trials. We expect trial volume to increase by 20% which will require this experiment to run for 30 days to reach significance. We will always want to monitor the rate at which gold trials upgrade to ensure that does not decrease by more than 5% as that could offset any gains in IACV from Bronze or Silver upgrades (source calculations used for predicted increase and duration).

Experiment design & implementation

This experiment will be broken out by 50% of free users will receive the control and 50% of free users will receive the experiment. The control group will get the standard experience that exists today where they only have one option at the top of the page to start a gold trial. Our experiment group will have trial options within each tier in the billing page next to the associated upgrade buttons for each tier.

This experiment will run on the newly designed group billing page #31471

This first iteration of this experiment will direct the user in the experiment group that select a Bronze or Silver trial into the trial process for a Gold trial. We realize this is not a ideal user experience but it will allow us to learn quickly and decide if we should build out the ability for users to trial Bronze and Silver.

ICE score

Impact Confidence Ease Score
6 4 6 5.33

Tracking

We will need to track the following:

Control group

  • Number of views to the group billing page
  • Number of users that start Gold trials
  • If a trial does upgrade, what tier do they upgrade to, how many seats do they purchase
  • Number of users that select a "buy now" option and upgrade along with the tier and number of seats

Experiment group

  • Number of views to the group billing page
  • Number of users that start Bronze, Silver and Gold trials
  • If a trial does upgrade, what trial option did they select what tier do they upgrade to, how many seats do they purchase
  • Number of users that select a "buy now" option and upgrade along with the tier and number of seats

Known assumptions

As noted in the expected outcomes section we will closely watch the rate at which users select gold trials and upgrade to ensure we’re not negatively affecting the total IACV.

Results, learnings, next steps

TBD

Edited by Sam Awezec