## Experiment summary We believe that when we present a trigger to a user that encourages the next most logical step for them to take in the DevOps process they will be willing to try and experiment with more of our product and features. To verify that, we will create a growth loop that includes a [trigger](https://gitlab.com/groups/gitlab-org/-/epics/2112) (encouraging the action) and a [delighter](https://gitlab.com/gitlab-org/growth/ui-ux/issues/5) (celebrating the completion of a task) that will draw the attention of a user, encouraging them to try another feature. When that feature has been used and a task has been completed we will present them with a delighter that will hopefully create a memorable experience and encourage the continued behavior of using the suggested feature. And we’ll measure the impact by exposing a suggestion to a small sample of users and if they complete the growth loop we will consider this a win. If they do not complete the growth loop we will consider it a loss. ## Status Active on .com only for 100% of users ## Artifacts [Walkthrough](https://youtu.be/511me3_UOIE) [Prototype of the flow](https://www.sketch.com/s/1794d37d-c722-4d32-862e-9c6c5d831149/a/zn1Z9o/play) [Dashboard w/ Funnel Reporting for this experiment](https://app.periscopedata.com/app/gitlab/658519/WIP:-Growth---Expansion---Experiment---MR-with-no-Pipelines) [CI Minutes Dashboard](https://app.periscopedata.com/app/gitlab/434763/CI-Minutes) [Experiment Results Tracking Issue](https://gitlab.com/gitlab-org/gitlab/-/issues/212896)  *With this experiment, we're trying to increase users motivation and ability to create a pipeline. Based on FOGG behaviour model, if both are sufficient, the trigger is above the threshold of where the desired behaviour happens.* ## Hypothesis <!-- The hypothesis represents the high-level thought process in creating the experiment but does not need to be proven in one experiment. For example, you could have a hypothesis that “users would benefit from more easily being able to start a trial” and your first experiment could fail, that doesn’t void your hypothesis only indicates you may need to think of a new iterative experiment that would still align with your hypothesis. --> More users will create pipelines if they were introduced to them when they have merge requests. ## Personas * [Devon (DevOps Engineer)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#devon-devops-engineer) * [Sasha (Software Developer)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#sasha-software-developer) ## Business problem <!-- Where the hypothesis is focused on the user/customer, the business problem represents why/how an experiment in this area could positively impact the business. For example, trials represent a significant way for GitLab to produce valuable leads for the sales team. --> Customers are not aware of how easy it can be to set up a pipeline and use ci minutes to deploy code today. We want to make that experience easy for the user and help them reach that ah-ha moment quickly. ## Supporting data <!-- Why should we run this experiment? What’s the potential impact? Show supporting data that’s both qualitative and quantitative. Quantitative example, we generate 30,000 sign ups a month and 900 trails within 90 days (3%) with a close rate of 10% and an IACV of $400. If we’re able to increase our trial volume by 10% percent (990 trials a month) we will generate an additional $3,600 IACV if our close rates remain constant. Qualitative example, in searching Zendesk I was able to find 10 support tickets in the last 30 days that referenced difficulties with starting a trial due to the user not being an admin. (all numbers are hypothetical and only listed for the purpose of having an example) --> ## Experiment design & implementation <!-- What is the experiment we’re going to run? How long do you believe it will need to run to reach significance? For example, our experiment would be to allow non-admins to request a trial through their admin, to detect a 10% change from our baseline conversion rate we’ll need a sample size of 57,000 (source Optimizely), with our current sign up rate of 30,000 a month this experiment will need to run for ~2 months. (all numbers are hypothetical and only listed for the purpose of having an example) --> ### Experiment design ##### Duration: * This will be ongoing until we reach statistical significance (we started with 2 weeks and are increasing exposure every week) ##### Conversion: * If more than 15% of users who engage, complete the growth loop we will consider this a victory ##### Feature flag: * Let's use a feature flag so we have the control in place to toggle on and off * Let's also start by exposing this to 10% of users on gitlab.com on a merge request where there is no pipeline. If we need more to reach statistical significance we can increase it as we go. ### Implementation (proposed MVC) By leveraging the [JTBD framework](https://jtbd.info/) coupled with the hook model we’re proposing the following for our MVC ### Summary: In efforts to increase pipeline activation for users who have merge requests, we are going to call attention to not only feature activation, but we will also tell a story with contextual nudges that explains why activating a pipeline is valuable. This will be done with a series of in-app messages starting from the merge request screen. The first message will articulate the benefits of using pipelines with a call to action. This will then bring them to the next message where they can add a file to create a pipeline. When a user advances, we will navigate them to the page where the user can then create a file to run pipelines. On this page, we will call attention to the templates section so the user can quickly see that they don’t have to start from scratch. (If the data is available we could automatically suggest a template or order the list so their file template would be on top). Once the user has selected a template, we will then nudge the user to commit changes with a message that provides reassurance that this is the right thing to do, reinforcing the value pipelines will provide. When this is completed, we will then celebrate the effort with a delightful message. The message with being fun, exciting, and tasteful by design, providing them with a memorable experience and prompting good feelings about their decision to create a pipeline. ### So what is the job to be done? > When I'm creating a merge request, I want the pipeline to run tests and check that I'm not adding technical debt or vulnerabilities to the code so that I can avoid future code refactoring and potential security problems. ### How we will apply the [hook model](https://fs.blog/2014/03/hooked/) to this issue: #### Trigger (external/internal) * When a **merge request is created**, this will act as our external trigger to nudge users to set up a pipeline. * The **concern of tech debt and vulnerabilities** will act as our internal trigger to incentivize the user to set up a pipeline. (we'll need to communicate this effectively so it does not draw too much from our fear/hope heuristic) #### Action * Guided pipeline set up nudges will drive the actions needed to create a pipeline. #### Reward * The value of the pipeline is the reward. Pipelines reduce tech debt and vulnerabilities. We'll celebrate that with a congratulations message to tie in the [peak end rule](https://en.wikipedia.org/wiki/Peak%E2%80%93end_rule). #### Investment * Creating the pipeline is the investment. The heuristic of [sunk cost effect](https://www.sciencedirect.com/science/article/abs/pii/0749597885900494) kicks in after a pipeline is created. ### What does the flow look like?  [Prototype of the flow](https://sketch.cloud/s/GYZAD/a/zn1Z9o/play) ### Tracking Requirements We'll need to create a funnel with aggregate data that will give us a sense of drop-offs and completions * Track how many times we show nudge #1 * Track how many time we show nudge #2 * Track how many times we show nudge #3 * Track how many times we show nudge #4 * Track how many times each button is clicked on nudge #1 * Track how many times the go to Pipeline button is clicked on nudge team-tasks#43 * Collect the template selected when a user is shown nudge #_2 ## ICE score | Impact | Confidence | Ease | Score | | ------ | ------ | ------ | ------ | | 7 | 9 | 4 | 6.6 | ## Known assumptions <!-- This is an area to call out known assumptions in the experiment, this is especially helpful for any future colleagues that join the team so they understand other potential influences and how they were accounted for. This section is also helpful in framing possible scenarios and to keep the door open for the next steps. For example, we’re hoping our experiment will increase the number of people that start a trial but we’re assuming the conversion rate to paid and IACV will remain the same. This is a known assumption and depending on the results of the experiment could impact the direction we take on any future iterations. --> * Users have MRs and no pipelines * Users are on a plan that allows them to activate pipelines * Exclude users who have pipelines set up ## Results, learnings, next steps <!-- What were the results of the experiment? Was the experiment a success or a failure? Based on the results should we remove the code or advocate that it become a permanent part of the experience for all users? Are there future experiments the team is going to run based on these results (include a link to a new issue)? For example, our trial experiment was successful we increased the trial create rate by 10% but we saw a 1% drop in our close rate which means our net impact on IACV was negative $360 (990 * 0.09 * 400 compared tot he control of 900 * 0.1 * 400). Our next experiment (link) will focus on increasing the value once a user starts a trial. (all numbers are hypothetical and only listed for the purpose of having an example) --> **Checklist** * [x] Fill in the experiment summary and write more about the details of the experiment in the rest of the issue description. Some of these may be filled in through time (the "Result, learnings, next steps" section for example) but at least the experiment summary should be filled in right from the start. * [x] Add the label of the Growth subgroup that will work on this experiment. * [x] Mention the Product Manager and at least one Product Designer from the group that owns the part of the product that the experiment will affect. * [x] Fill in the values in the [ICE score table](#ice-score) ping other team members for the values you aren’t confident about (i.e. engineering should almost always fill out the ease section). Add the product~12127129 label to indicate that the score is incomplete. * [x] Replace the product~12127129 with an ICE low/medium/high score label once all values in the ICE table have been added. * [x] Mention the [at]gitlab-core-team team and ask for their feedback. ## Follow on experiments and issues (additional ideas that came out of this epic) * Email campaign to increase conversion of pipeline activations * Nudge #1 copy variants * Nudge #2 copy variants * Nudge #3 copy variants * Nudge #4 copy variants * Create in-app nudge to suggest learning more about ci minutes and tier upgrades for users who set up a pipeline * Email campaign to suggest using ci minutes and education about plans with more minutes * Expand target audience from paying only and include free as well cc. @pcalder @jackie_fraser @dstull @matejlatin