<!-- Title suggestion: Experiment Implementation: [description] --> # Experiment Summary We believe we can increase paid feature discovery and paid conversion by creating Discovery pages for each stage. This experiment will be our MVC effort to test this cross-stage feature discovery concept. If proven successful, we will look to expand this discovery experience across multiple stages in the product. # Hypothesis <!-- The hypothesis represents the high-level thought process in creating the experiment but does not need to be proven in one experiment. For example, you could have a hypothesis that “users would benefit from more easily being able to start a trial” and your first experiment could fail, that doesn’t void your hypothesis only indicates you may need to think of a new iterative experiment that would still align with your hypothesis. --> We believe by increasing discoverability of paid features we can increase the paid conversion rate. # Design <!-- This should include the contexts that determine the reproducibility (stickiness) of an experiment. This means that if you want the same behavior for a user, the context would be user, or if you want all users when viewing a specific project, the context would be the project being viewed, etc. --> ## First iteration of the landing page #### Entry location The user enters this experiment (their namespace is assigned a variant) when we render the `question_mark` menu (a.k.a. the “Help” menu or the `?` menu) in the top horizontal nav. At the moment of render, the variant assignment will determine if the “Discover Premium & Ultimate” menu item is added to the menu (`candidate`) or not (`control`).  #### Desktop <details><summary>Click to expand</summary>  </details> #### Mobile <details><summary>Click to expand</summary> <img src="/uploads/5aa9c2707409d296020884132ea544c4/landing-page-mobile.gif" width="250"> </details> #### Flow Start a trial <details><summary>Click to expand</summary> <img src="/uploads/c8d738f27f1bb0b608a34de17cf30080/start-trial-landing.gif" width="450"> </details> #### Flow Contact Sales <details><summary>Click to expand</summary> <img src="/uploads/c12661f5d7eea8e4bbb08c593636ff71/landing-contact-sales.gif" width="450"> </details> ## Logo & Case studies approval [See this thread](https://gitlab.com/gitlab-org/gitlab/-/issues/343177#note_806827505) ### [Design specs](https://www.figma.com/file/O3ZoGVX93vYoFO2oRcCqD9/Admin-Feature-Discovery?node-id=215%3A12862) # Rollout strategy <!-- This is currently called A/B test, which isn't accurate for multi-variants. Let's call this rollout strategy. It should outline the percentages for variants and if there's more than one step to this, each of those steps and the timing for those steps (e.g. 30 days after initial rollout). --> Experiment design: - Control: Standard experience without the discovery page - Candidate: Candidate group will be able to access and view the discovery page from the Help Menu (`?`) dropdown Implementation: * 80/20 (`control`/`candidate`) to start, then roll to 50/50 once data is validated # Inclusions and exclusions <!-- These would be the rules for which given context (and are limited to context or resolvable at experiment time details) is included or excluded from the test. An example of this would be to only run an experiment on groups less than N number of days old. --> Inclusions: * Owner/admin members of namespaces which are on the Free tier Exclusions: * All non-owner/admin members of all namespaces * All members of namespaces which are not on the Free tier **Note:** Users can be members of multiple namespaces, so they may see the experiment when in the context of one namespace and not when in the context of another. # Segmentation <!-- Rules for always saying context with these criteria always get this variant. For instance, if you want to always give groups less than N number of days old the experiment experience, they are specified here. This is different from the exclusion rules above. --> None. # Success Metrics & Experiment Assignment [for Product Analytics] ## Primary Success metric - Free to paid conversion rate ## Secondary metric - Rate at which a primary action at the namespace level is taken with a primary action being defined as starting a trial, upgrading to a paid subscription, submitting a hand-raise PQL ## Experiment Assignment & notes - We will assign [control or candidate] this experiment at the **namespace** level - We will pass assignment [control or candidate] to each individual event # Tracking Details ## Snowplow tracking - [gitlab_experiment context json schema](https://gitlab.com/gitlab-org/iglu/-/blob/master/public/schemas/com.gitlab/gitlab_experiment/jsonschema/0-3-0) - the `gitlab_experiment` context is automatically applied to every experiment-related event - `namespace` and `user` as part of context `key` property, experiment will be sticky to the `namespace` - [gitlab_standard context json schema](https://gitlab.com/gitlab-org/iglu/-/blob/master/public/schemas/com.gitlab/gitlab_standard/jsonschema/1-0-8) - the `gitlab_standard` context is automatically applied to every event - will contain `user_id`, `namespace_id`, & `project_id` (when available) - see [taxonomy](https://docs.gitlab.com/ee/development/snowplow/index.html#structured-event-taxonomy) for a guide. - `category` will default to the page we are on (i.e. `body.dataset.page`), except in the case of the `assignment` event where the `category` will be the experiment name: `fdm_deliver_software_faster`. | action | label | Notes | | -------- | ------ | ----- | | `assignment` | | This event will automatically happen _every time_ we render a page in a context where the user is eligible for the experiment. | | `click_question_mark_link` | `main_navigation` | A pre-existing event that we’re attaching experiment context to which fires for all users when they open the `?` dropdown. **Note:** Only participants in our experiment (whether in `control` or `candidate`) will have the additional `gitlab_experiment` context attached. | | `click_link` | `cross_stage_fdm` | Event only fires for users in the `variant` that select our new link to the cross stage feature discovery page we've created for this test located within the `?` dropdown | | `render` | `cross_stage_fdm` | Event only fires for users in the `variant` that view the cross stage feature discovery page we've created for this test | | `click_button` | `start_trial` or `hand_raise_PQL` | Event only fires for users in the `variant` that click links within the cross stage feature discovery page we've created for this test either selecting "start a free trial" or "contact sales". Note we should capture this event whether they select the CTAs at the top of the page or the same CTAs located at the bottom. If a user selects both CTAs "start a free trial" and "contact sales" then we should fire two events. | ## Core actions tracking If a user selects to "start a trial" or "contact sales" then we want to ensure these actions are tracked. These values should be passed to the `leads` table utilizing the following values: | Location/CTA | GLM_source | GLM_content | | ------ | ------ | ------ | | "start trial" | gitlab.com | cross_stage_fdm | | "contact sales | gitlab.com | cross_stage_fdm | - this will merely mean a param difference when clicking one of the buttons, which will eventually get populated in CustomersDot when the lead is submitted. ### Event Tracking - Screenshot & [Loom Video] ## Development Updates - <details><summary><code>2022-01-18</code>: Breaking down my perception of all the work needing to be done:</summary> ### Core engineering work - [x] Create the new experiment feature flag - [x] Use the new feature flag to render (or not) a new menu item - [x] Create a new route, controller, action, & view for the feature discovery dashboard & hook it into the new menu item - [x] Do initial pass of the new dashboard’s contents (text & minimal design, basically) - [x] Add the visual elements (images, like the SVGs) - [ ] Make the design work for [the mobile version of the view](https://gitlab.com/gitlab-org/gitlab/-/issues/348494#mobile) (i.e. the horizontally scrolling feature sections) - [x] Make the design work for [the desktop version of the view](https://gitlab.com/gitlab-org/gitlab/-/issues/348494#desktop) (i.e. a grid of cards which _may_ need to all be the same height, even with translations) - [ ] Test out the view in supported browsers (wider for desktop, narrower for mobile) - [x] Hook up the [“Start a free trial” CTAs to the trial creation flow](https://gitlab.com/gitlab-org/gitlab/-/issues/348494#flow-start-a-trial) - [x] Hook up the [“Contact Sales” CTAs to the contact sales form modal](https://gitlab.com/gitlab-org/gitlab/-/issues/348494#flow-contact-sales) - [x] [Add records to the `leads` table](https://gitlab.com/gitlab-org/gitlab/-/issues/348494#core-actions-tracking) for each of the CTAs (might need to make an API call in the case of the contact sales form modal) - [x] [Add tracking](https://gitlab.com/gitlab-org/gitlab/-/issues/348494#snowplow-tracking) to the new menu link & to the CTAs on the dashboard page - [ ] ([Maybe?](https://gitlab.com/gitlab-org/gitlab/-/issues/348494#note_814385000) Add tracking to the trial creation flow) - [ ] ([Maybe?](https://gitlab.com/gitlab-org/gitlab/-/issues/348494#note_814385000) Add tracking to the contact sales form modal submissions) ### Additional tasks - [ ] Get all parts reviewed - [ ] Implement reviewer feedback - [ ] Get all parts approved - [ ] Wait for all parts to be merged - [ ] Wait for all parts to be deployed to staging - [ ] Review all parts on staging - [ ] Wait for all parts to deploy to production - [ ] Launch the experiment </details> - <details open><summary><code>2022-01-25</code>: Breaking it down into more detail & vertically sliced iterations</summary> ## Vertically sliced iterations 1. <details><summary><strong>✅ Smallest & simplest MVC possible</strong></summary> - [x] Add experiment feature flag: `cross_stage_fdm` - [x] Use experiment feature flag to add menu item to the Help (a.k.a. `?`) menu - [x] Limit the experiment to owners/admins of free namespaces - [x] New menu item takes user to our feature discovery moment page - [x] Track when a user opens the `?` menu (both variants) - [x] This is already happening, we just need to add our experiment context to that tracking: `experiement=cross_stage_fdm` - Current tracking is: `action=click_question_mark_link`, `label=main_navigation`, `property=navigation` - [x] Track when a user clicks on the new menu item (candidate only) - [x] Track render (as an FE event) of feature discovery page - [x] Create MVC version of the feature discovery page: - [x] Choose an approrpriate layout (or create a new one) - [x] Title: “Discover Premium & Ultimate.” - [x] Copy: “Access advanced features, build more efficiently, strengthen security and compliance.” - [x] Add the “Start a free trial” CTA - [x] Add the “Start a free trial” button to the feature discovery page - [x] Add the desired `glm` params to the button: `glm_source=gitlab.com&glm_content=cross_stage_fdm` - [x] Verify that these params make it all the way through to the `leads` table in CustomersDot - [x] Track clicks of the “Start a free trial” CTA button - [x] Add the “Contact sales” CTA - [x] Add the “Contact sales” CTA button to the feature discovery page - [x] Pull in the necessary JS for the `hand_raise_lead_button.vue` component to work - [x] Track clicks of the button - [x] Figure out how to get the desired `glm` params to be submitted with the form: `glm_source=gitlab.com&glm_content=cross_stage_fdm` - [x] Verify that these params make it all the way through to the `leads` table in CustomersDot - [x] Promo image (the one taken from the `about.gitlab.com/enterprise` page) - [x] Add tests for all the new things - [x] A “happy path” feature spec (follows the flow of the experiment ensuring the menu item exists, the page renders, and the buttons exist & are clickable) - [x] Helper specs (testing new helper) - [x] Routing specs (for the new route) - [x] View specs (testing the contents of the view) - [x] New dashboard view - [x] New help menu partial - Can only do a “happy path” test. If we try to test the non-rendering cases, we hit one of two issues: - `Error in ./ee/spec/views/layouts/header/help_dropdown/_cross_stage_fdm.html.haml_spec.rb:34: no implicit conversion of nil into String` - `ee/app/views/layouts/header/help_dropdown/_cross_stage_fdm.html.haml:3 [W] RuboCop: Lint/TopLevelReturnWithArgument: Top level return with argument detected.` - [x] Request specs (testing new controller & action) - [x] Update JS specs for hand_raise_lead_button.vue </details> 1. <details><summary><strong>✅ Add the “Speed. Efficiency. Trust.” section</strong></summary> - [x] Add the title: “Speed. Efficiency. Trust.” - [x] Create data objects for the three cards - [x] Figure out how to pull in the SVG icons - [x] Style it up - [x] Update the view specs </details> 1. <details><summary><strong>✅ Add the “Access advanced features.” section</strong></summary> - [x] Add the title: “Access advanced features.” - [x] Create data objects for the 10 cards - [x] Use the same method for pulling in the SVG icons - [x] Style it up - [x] Update the view specs </details> * :point_up: Above is [the slice we will deliver for the MVC for 14.8](https://gitlab.com/gitlab-org/gitlab/-/issues/348494#note_820261248) :point_up: * :point_down: Below is remainder of work that [could be scheduled for 14.9 or can wait for experiment cleanup](https://gitlab.com/gitlab-org/gitlab/-/issues/348494#note_843765365) :point_down: 1. <details><summary><strong>Add the semi-sticky footer</strong></summary> - [ ] Add a footer to the layout (might need to create your own at this point) - [ ] Make it semi-sticky to the viewport (i.e. if the content is less than the height of the viewport, the foot sticks to the very bottom of the viewport, if the content is greater than the height of the viewport, the footer flows with the content) - [ ] Add the footer title: “Free 30 days trial” - [ ] Add the footer copy: “Try everything for free. No credit card required.” - [ ] Reproduce the CTAs from the header into the footer - [ ] Double-check that it works well to have multiple versions of the `hand_raise_lead_button.vue` component/app on the page - [ ] Update the view specs - [ ] Update the feature specs to click on CTAs in the header & in the footer to make sure both places work as expected </details> 1. <details><summary><strong>Add social proof sections</strong></summary> - [ ] Add the copy to the header area: “Trusted by enterprises, loved by developers” - [ ] Add the chosen logos to the header: hotjar, siemens, wish, & KnowBe4 - [ ] Add the “How they use GitLab.” section - [ ] Add the title: “How they use GitLab.” - [ ] Create data objects for the three examples - [ ] Figure out where/how to get the color versions of the logos (this might be the default, actually) - [ ] Style it all up - [ ] Update the view specs </details> 1. <details><summary><strong>Create special styles for the mobile version of the “Access advanced features.” section</strong></summary> - [ ] We want a single row which scrolls horizontally - [ ] Test & debug the horizontal scrolling effect in multiple browsers </details> 1. <details><summary><strong>Consider opening up the experiment to owners/admins of trial namespaces</strong></summary> - [ ] [Figure out what to do with the “Start a free trial” CTA](https://gitlab.com/gitlab-org/gitlab/-/issues/348494#note_815850652) - [ ] Adjust the current restriction to also allow owners/admins of trial namespaces - [ ] Change the “Start a trial” CTA wording, URL, & tracking if the namespace is on a trial - [ ] Change wording to something like “Upgrade to Ultimate” - [ ] Change URL to point to the namespace’s billing page - [ ] Change tracking label to be something like `upgrade` - [ ] Update all associated tests, including adding new tests for this new scenario </details> 1. <details><summary><strong>Consider opening up the experiment to owners/admins of all non-Ultimate namespaces</strong></summary> - If we change the “Start a trial” CTA to something like “Upgrade to Ultimate”, then we might as well open up the experiment to owners & admins of all non-Ultimate namespaces - [ ] Remove the restriction to be an owner/admin of a free or trial namespace - [ ] Add a restriction to be an owner/admin of a non-Ultimate namespace - [ ] Reuse the same “Upgrade to Ultimate” (or whatever) alternate CTA for non-free-namespace users - [ ] Update all associated tests, including adding new tests for the new scenario </details> </details> - `…`