[Engineering] Experiment requiring group trials
Experiment summary
We believe that we can increase the ARR generated from SaaS trials by only allowing trial sign ups to start a group trial.
To verify that, we will run an A/B experiment where 50% of traffic can start a personal or group trial and 50% will be required to create a group for the trial.
And we’ll measure the impact on total ARR generated between the two groups along with the trial close rates for both groups
Hypothesis
We believe that when some users start a trial they are unaware of the difference between a personal and group trial and unknowingly select the wrong option.
Business problem
A personal trial has a maximum ARR value of one paid seat whereas the value of a group trial is much large due to the infinite number of possible users within the group that could be included within an upgrade. Also, if a user is unaware of the differences between the trial types and selects a personal trial when they're hoping for their team to trial the features only the admin will gain access during the trial reducing the likelihood that the account will upgrade.
Supporting data
We've found much higher upgrade rates for group trials than we do for personal trials, as displayed in (internal link)this dashboard. Part of this is likely due to the persona that is starting a personal trial over a group trial but we believe some if it is related to users not understanding a group trial, is required to ensure all users within an organization have access to the trial.
Expected outcome
We expect by requiring group creation for a trial, we will generate a net increase of 5 percentage points in the combined trial upgrade rate vs the control. We also believe we will see more total ARR generated in the experiment group vs the control.
This experiment is being tracked here.
Experiment design & implementation
50% of the trial flow will have the experience today where they can select to apply the trial to their personal username, existing group or create a new group. The experiment group will only see options to either apply the trial to an existing group or create a new group if they already have existing groups. If the user in the experiment group does not yet have a group they will be prompted to create one.
Control experience (default experience today) | Experiment group experience if they already have a group | Experiment group experience if do not yet have a group |
---|---|---|
https://gitlab.com/-/trials/select | https://gitlab.com/-/trials/select | https://gitlab.com/-/trials/select |
ICE score
Impact | Confidence | Ease | Score |
---|---|---|---|
8 | 8 | 7 | 8.333 |
Known assumptions
We believe that some personas are selecting personal trials knowing they will never upgrade but we currently have no way to quantify the number without running this experiment.
Results, learnings, next steps
This experiment has reached 96% significance with the experiment winning out over the control. A full review on the impact of this experiment on the trial conversion rate can be found here (note this is an internal issue as it discusses actual conversion rates).
Checklist
-
Fill in the experiment summary and write more about the details of the experiment in the rest of the issue description. Some of these may be filled in through time (the "Result, learnings, next steps" section for example) but at least the experiment summary should be filled in right from the start. -
Add the label of the Growth subgroup that will work on this experiment. -
Mention the Product Manager and at least one Product Designer from the group that owns the part of the product that the experiment will affect. -
Fill in the values in the ICE score table ping other team members for the values you aren’t confident about (i.e. engineering should almost always fill out the ease section). Add the ICE Score Needed in GitLab.org / Growth / Product label to indicate that the score is incomplete. -
Replace the ICE Score Needed in GitLab.org / Growth / Product with an ICE low/medium/high score label once all values in the ICE table have been added.