## Summary New signups don't immediately understand what features are paid and included in their 30-day Ultimate Trial. We believe that if we promoted these paid features more, more would adopt them and convert to paid. Most trials land on Learn GitLab after signup. We believe we can leverage Learn GitLab as a means to continuously promote paid features during the trial period. To verify that, we will test an alternative 'promotion' of paid features on Learn GitLab. We will be measuring the impact of this to adoption of Ultimate paid features (ex. Security). Additionally we will observe impacts to Team Activation and paid conversion. Relates to Q4 KR https://gitlab.com/gitlab-com/gitlab-OKRs/-/work_items/5395 ## Hypothesis <!-- The hypothesis represents the high-level thought process in creating the experiment but does not need to be proven in one experiment. For example, you could have a hypothesis that “users would benefit from more easily being able to start a trial” and your first experiment could fail, that doesn’t void your hypothesis only indicates you may need to think of a new iterative experiment that would still align with your hypothesis. --> We believe that new signups are not adopting paid features during trial because they don't know what features are paid vs free. Prediction: We believe that if we promoted these paid features more, more would adopt them and convert to paid. ## Business problem <!-- Where the hypothesis is focused on the user/customer, the business problem represents why/how an experiment in this area could positively impact the business. For example, trials represent a significant way for GitLab to produce valuable leads for the sales team. --> A [previous experiment](https://gitlab.com/groups/gitlab-org/-/epics/10262) increased the volume of trial signups as well as 2nd user adds. While increasing the # of trials instead of free is advantageous for our business, we need to invest more in the product experience to ensure signups are realizing value and converting to paid. To start we will be investing in the first product experiences (ex. Learn GitLab) and exploring solutions that better help new signups adopt and activate. ## Supporting data <!-- Why should we run this experiment? What’s the potential impact? Show supporting data that’s both qualitative and quantitative. Quantitative example, we generate 30,000 sign ups a month and 900 trails within 90 days (3%) with a close rate of 10% and an IACV of $400. If we’re able to increase our trial volume by 10% percent (990 trials a month) we will generate an additional $3,600 IACV if our close rates remain constant. Qualitative example, in searching Zendesk I was able to find 10 support tickets in the last 30 days that referenced difficulties with starting a trial due to the user not being an admin. (all numbers are hypothetical and only listed for the purpose of having an example) --> Insert 1st Secure SMAU adoption ## Expected outcome <!-- What is the expected outcome of this experiment, what metric are we trying to move? Are there any metrics we know we do not want to impact? For example, we want to impact IACV by increasing the rate at which users start trials within 30 days but we also want to ensure we don't increase the churn rate for users who've recently purchased. --> Experiment Results will be tracked on our [Filterable Growth Experiment Analysis Dashboard ](https://app.periscopedata.com/app/gitlab/1121391/Filterable-Growth-Experiment-Analysis) #### Primary Success Metric 1st Secure SMAU adoption #### Secondary Success Metric - Team Activation #### Other Metrics to track - Paid Conversion #### Guardrail Metic TBD #### Engagement Metrics No change to event tracking ## Experiment design & implementation <!-- What is the experiment we’re going to run? How long do you believe it will need to run to reach significance? For example, our experiment would be to allow non-admins to request a trial through their admin, to detect a 10% change from our baseline conversion rate we’ll need a sample size of 57,000 (source Optimizely), with our current sign up rate of 30,000 a month this experiment will need to run for ~2 months. (all numbers are hypothetical and only listed for the purpose of having an example) --> ## Implementation Details in [ENG issue](https://gitlab.com/gitlab-org/gitlab/-/issues/424506) ## Known assumptions/risks <!-- This is an area to call out known assumptions in the experiment, this is especially helpful for any future colleagues that join the team so they understand other potential influences and how they were accounted for. This section is also helpful in framing possible scenarios and to keep the door open for the next steps. For example, we’re hoping our experiment will increase the number of people that start a trial but we’re assuming the conversion rate to paid and IACV will remain the same. This is a known assumption and depending on the results of the experiment could impact the direction we take on any future iterations. --> ## Results, lessons learned, next steps <!-- What were the results of the experiment? Was the experiment a success or a failure? Based on the results should we remove the code or advocate that it become a permanent part of the experience for all users? Are there future experiments the team is going to run based off these results (include a link to new issue)? For example, our trial experiment was successful we increased the trial create rate by 10% but we saw a 1% drop in our close rate which means our net impact on IACV was negative $360 (990 * 0.09 * 400 compared tot he control of 900 * 0.1 * 400). Our next experiment (link) will focus on increasing the value once a user starts a trial. (all numbers are hypothetical and only listed for the purpose of having an example) --> ## Checklist * [x] Fill in the experiment summary and write more about the details of the experiment in the rest of the issue description. Some of these may be filled in through time (the "Result, learnings, next steps" section for example) but at least the experiment summary should be filled in right from the start. * [x] Add the label of the `group::` that will work on this experiment (if known). * [x] Mention the Product Manager, Engineering Manager, and at least one Product Designer from the group that owns the part of the product that the experiment will affect. * [x] Mention the [at]gitlab-core-team team and ask for their feedback.