Conversion Experiment: allow users to start premium trials

Experiment summary

We believe that we can increase trial volume and total upgrades by allowing users to start Premium as well as Ultimate trials. To verify this, we will allow a cohort of users in-app to start both Premium and Ultimate trials. We will monitor the page view to trials started and upgrades in the experiment vs the control.

Hypothesis

By allowing users to select the trial they're most interested in we'll increase the total ARR we generate from trials.

Business problem

We know that the majority of the Ultimate trials today that do upgrade pick a Premium plan therefore we believe we have a chance to increase overall trial volume and net ARR by allowing users the opportunity to start a Premium trial.

Supporting data

Expected outcome

We believe we will see a larger number of trials started by offering a Premium trial and in the end net out more ARR for the business by offering users the ability to pick the trial that best suits their needs.

Experiment design & implementation

This experiment will be run on the in-app billing page. Users in the control will get the same page that is live today where they can choose to either upgrade, talk to sales or start an Ultimate trial. Users in the experiment cohort will have options to upgrade, talk to sales, start an Ultimate or Premium trial.

Control Experiment
image image

Design Specs - (waiting on link)

Experiment tracking

To track this experiment we should do the following:

  • Add the group_ids of namespaces in the control or experiment to the experiment_users table
  • Add the following frontend events to the in-app billing page both for the control and experiment, the event context should contain if the users is in the control or experiment
    • Billing page load
    • CTA select with a value stored for which CTA was selected i.e. Upgrade (note for Ugrade we should also store what tier they selected i.e. Premium or Ultimate) Contact sales Start a trial (note we should also capture what tier they selected in the control the value will always be Ultimate but in the experiment value can be Ultimate or Premium

ICE score

Impact Confidence Ease Score
value 1 value 2 value 3 Average(1:3)

Known assumptions

We know that providing users with the ability to start a Premium trial will reduce the Ultimate trial signup rate. We will monitor the total ARR generated from the control vs experiment groups to understand which experience nets more revenue for the business.

Results, lessons learned, next steps

TBD

Checklist

  • Fill in the experiment summary and write more about the details of the experiment in the rest of the issue description. Some of these may be filled in through time (the "Result, learnings, next steps" section for example) but at least the experiment summary should be filled in right from the start.
  • Add the label of the group:: that will work on this experiment (if known).
  • Mention the Product Manager, Engineering Manager, and at least one Product Designer from the group that owns the part of the product that the experiment will affect.
  • Fill in the values in the ICE score table ping other team members for the values you aren’t confident about (i.e. engineering should almost always fill out the ease section). Add the ~"ICE Score Needed" label to indicate that the score is incomplete.
  • Replace the ~"ICE Score Needed" with an ICE low/medium/high score label once all values in the ICE table have been added.
  • Mention the [at]gitlab-core-team team and ask for their feedback.