For free users of GitLab.com test the demand for trials of bronze and silver

Experiment summary

We believe that by providing users the option to trial Bronze and Silver along with Gold we will increase the total IACV we generate from trials.

To verify that, we will run an experiment where a cohort of free users has the option to trial Bronze, Silver and Gold and compare the IACV to the control where users can only trial Gold.

Hypothesis

We believe that we can increase the amount of IACV generated from trials by offering trials of all tiers (Bronze, Silver, Gold).

Business problem

Currently, users can only start a Gold trial, we may be losing out on trial demand from users that are only interested in Bronze or Silver.

Supporting data

Today users can only trial Gold, in periscope we found that of all the trial upgrades in the last 3 months, 7.07% upgrade during or after the trial. Of that 7.0% that upgraded, 6.22% purchased Gold, 39.74% purchased Silver and 54.03% purchased Bronze (source).

Expected outcome

We expect that by increasing the overall volume of trials by allowing users to select trials of Bronze and Silver we will increase the total IACV generated compared to only allowing users to pick gold trials. We expect trial volume to increase by 20% which will require this experiment to run for 30 days to reach significance. We will always want to monitor the rate at which gold trials upgrade to ensure that does not decrease by more than 5% as that could offset any gains in IACV from Bronze or Silver upgrades (source calculations used for predicted increase and duration).

Experiment design & implementation

This experiment will be broken out by 50% of free users will receive the control and 50% of free users will receive the experiment. The control group will get the standard experience that exists today where they only have one option at the top of the page to start a gold trial. Our experiment group will have trial options within each tier in the billing page next to the associated upgrade buttons for each tier.

This experiment will run on the group billing page #31471 (closed)

Base Variant
image group-billing--2_2
Current version of group billing Users can test all tiers

ICE score

Impact Confidence Ease Score
6 4 6 5.33

Tracking

We will need to track the following:

Control group

  • Number of views to the group billing page
  • Number of users that start Gold trials
  • If a trial does upgrade, what tier do they upgrade to, how many seats do they purchase
  • Number of users that select a "buy now" option and upgrade along with the tier and number of seats
  • Trial CTA URL parameter
    • Gold: bsg-control-gold-trial

Experiment group

  • Number of views to the group billing page
  • Number of users that start Bronze, Silver and Gold trials
  • If a trial does upgrade, what trial option did they select what tier do they upgrade to, how many seats do they purchase
  • Number of users that select a "buy now" option and upgrade along with the tier and number of seats
  • Trial CTA URL parameters
    • Bronze trial CTA: bsg-exp-bronze-trial
    • Silver trial CTA: bsg-exp-silver-trial
    • Gold trial CTA: bsg-exp-gold-trial

Known assumptions

As noted in the expected outcomes section we will closely watch the rate at which users select gold trials and upgrade to ensure we’re not negatively affecting the total IACV.

Design

Proposed flow

  1. Users lands on the group clicks on "Try it for free" on Bronze or Silver card.
  2. User enters trial sign up flow, header of the flow changes to "bronze" or "silver" or "Gold" depending on what trial the user selected.
  3. User completes trial form and returns to their namespace with the trial tier they selected.

Internal action items:

  • Ensure support and sales is aware that we're going to a cohort of existing free users to trial Bronze, Silver, Gold.
  • Ensure the existing internal tools i.e. customers / license app correctly reference the trial tier Bronze, Silver, Gold.
  • Ensure Salesforce updates with the correct trial tier
  • Ensure marketing trial emails either remove the trial tier name or dynamically update the trial tier level in the associated copy
  • Ensure any in-app banners or trial messages correctly reference the correct trial tier.

Results, learnings, next steps

TBD

Edited by Kevin Comoli