Experiment Tracking: Trim-down the "skip copy" in the .com trial flow for non-GL users
Stopping the experiment feature flags
/chatops run feature delete trimmed_skip_trial_copy_experiment_percentage
Re-enabling the experiment
/chatops run feature set trimmed_skip_trial_copy_experiment_percentage 50
Testing
On gitlab.com or your local simply set the force experiment cookie
document.cookie = "force_experiment=trimmed_skip_trial_copy ; path=/";
Experiment summary
We believe that simplifying our copy is likely to clarify the process of completing the trial form, thus, increase the number of trial started.
To verify that, we will shorten the skip copy in the trial flow for users that are not authenticated to GitLab or entering the trial flow through the marketing website.
Hypothesis
Simplifying our copy is likely to clarify the process of completing the trial form, thus, increase the number of trial started. (speak-easy effect).
Business problem
We are seeing an increase in skipped trial for users that start a trial and are non-authenticated to GL. This flow is mainly relying on copy to convey the necessary steps to follow to start a trial. If the copy is ambiguous people are likely to abandon the process of starting a trial, thus impacts the number of trial started and IACV down the line.
Supporting data
~24.5% (combining all trial referring skip URL) of skip trial come from the trial pages (with an impressive 15% for the trial/new page). Source
Expected outcome
We will increase the successful trial signup rate.
Experiment design & implementation
We will remove the parenthesis part of the skip copy. The copy will then be "Skip trial" instead of "Skip trial" (continue with free account).
Control | Group |
---|---|
![]() |
![]() |
Current copy | Simplified copy |
Pages where this copy should be changed during the test:
- gitlab.com/-/trials/new
- gitlab.com/-/trials/select
Note: This test should only run for the control and experiment groups when the user originated from https://gitlab.com/-/trial_registrations/new
Experiment tracking
In order to determine the success or failure of this experiment, we need users enrolled into the experiment after they submit the form on https://gitlab.com/-/trial_registrations/new (or on the subsequent page load of gitlab.com/-/trials/new). The users should be added to the experiments table with a unique Experiment_ID
value, the users in the control should be given a Group_Type
value of 0
and the experiment group should get a Group_Type
value of 1
.
In order to track the success metric of successful trial signups, we need to fire a backend event associated with the user_id when the user completes the form on gitlab.com/-/trials/select where they select "Start your free trial". Or if the experiment table is ready we can add this as the success flag in the experiments table for this particular experiment.
Experiment rollout log tracking
Log tracking will be updated below in the comments
- Enabled #284956 (comment 499854957)
- ...
Experiment Results
This experiment is currently active, live results can be followed in this Sisense dashboard.
Roll Out Steps
-
Enable on staging -
Test on staging -
Enable on GitLab.com for individual groups/projects listed above and verify behavior. -
Announce on the issue an estimated time this will be enabled on GitLab.com -
Enable on GitLab.com by running chatops command in #production
-
Cross post chatops slack command to #support_gitlab-com
(more guidance when this is necessary in the dev docs) and in your team channel -
Announce on the issue that the flag has been enabled -
Remove feature flag and add changelog entry - a separate cleanup issue might be required -
After the flag removal is deployed, clean up the feature flag by running chatops command in #production
channel
Results:
This experiment was deemed a success on '2021-02-22' when it reached significance. We increased the trial signup rate by 12% with an absolute increase of 2%. (source)