Skip to content

ENG: MVC Create Cross-stage feature discovery moment

Experiment Summary

We believe we can increase paid feature discovery and paid conversion by creating Discovery pages for each stage. This experiment will be our MVC effort to test this cross-stage feature discovery concept. If proven successful, we will look to expand this discovery experience across multiple stages in the product.

Hypothesis

We believe by increasing discoverability of paid features we can increase the paid conversion rate.

Design

First iteration of the landing page

Entry location

The user enters this experiment (their namespace is assigned a variant) when we render the question_mark menu (a.k.a. the “Help” menu or the ? menu) in the top horizontal nav. At the moment of render, the variant assignment will determine if the “Discover Premium & Ultimate” menu item is added to the menu (candidate) or not (control).

CCSFDM_-Menu_Item-_discover_prem

Desktop

Click to expand

LandingPage-MVC-Desktop-discover-prem

Mobile

Click to expand

Flow Start a trial

Click to expand

Flow Contact Sales

Click to expand

Logo & Case studies approval

See this thread

Design specs

Rollout strategy

Experiment design:

  • Control: Standard experience without the discovery page
  • Candidate: Candidate group will be able to access and view the discovery page from the Help Menu (?) dropdown

Implementation:

  • 80/20 (control/candidate) to start, then roll to 50/50 once data is validated

Inclusions and exclusions

Inclusions:

  • Owner/admin members of namespaces which are on the Free tier

Exclusions:

  • All non-owner/admin members of all namespaces
  • All members of namespaces which are not on the Free tier

Note: Users can be members of multiple namespaces, so they may see the experiment when in the context of one namespace and not when in the context of another.

Segmentation

None.

Success Metrics & Experiment Assignment [for Product Analytics]

Primary Success metric

  • Free to paid conversion rate

Secondary metric

  • Rate at which a primary action at the namespace level is taken with a primary action being defined as starting a trial, upgrading to a paid subscription, submitting a hand-raise PQL

Experiment Assignment & notes

  • We will assign [control or candidate] this experiment at the namespace level
  • We will pass assignment [control or candidate] to each individual event

Tracking Details

Snowplow tracking

  • gitlab_experiment context json schema
    • the gitlab_experiment context is automatically applied to every experiment-related event
    • namespace and user as part of context key property, experiment will be sticky to the namespace
  • gitlab_standard context json schema
    • the gitlab_standard context is automatically applied to every event
    • will contain user_id, namespace_id, & project_id (when available)
  • see taxonomy for a guide.
  • category will default to the page we are on (i.e. body.dataset.page), except in the case of the assignment event where the category will be the experiment name: fdm_deliver_software_faster.
action label Notes
assignment This event will automatically happen every time we render a page in a context where the user is eligible for the experiment.
click_question_mark_link main_navigation A pre-existing event that we’re attaching experiment context to which fires for all users when they open the ? dropdown. Note: Only participants in our experiment (whether in control or candidate) will have the additional gitlab_experiment context attached.
click_link cross_stage_fdm Event only fires for users in the variant that select our new link to the cross stage feature discovery page we've created for this test located within the ? dropdown
render cross_stage_fdm Event only fires for users in the variant that view the cross stage feature discovery page we've created for this test
click_button start_trial or hand_raise_PQL Event only fires for users in the variant that click links within the cross stage feature discovery page we've created for this test either selecting "start a free trial" or "contact sales". Note we should capture this event whether they select the CTAs at the top of the page or the same CTAs located at the bottom. If a user selects both CTAs "start a free trial" and "contact sales" then we should fire two events.

Core actions tracking

If a user selects to "start a trial" or "contact sales" then we want to ensure these actions are tracked. These values should be passed to the leads table utilizing the following values:

Location/CTA GLM_source GLM_content
"start trial" gitlab.com cross_stage_fdm
"contact sales gitlab.com cross_stage_fdm
  • this will merely mean a param difference when clicking one of the buttons, which will eventually get populated in CustomersDot when the lead is submitted.

Event Tracking - Screenshot & [Loom Video]

Development Updates

  • 2022-01-18: Breaking down my perception of all the work needing to be done:

    Core engineering work

    Additional tasks

    • Get all parts reviewed
    • Implement reviewer feedback
    • Get all parts approved
    • Wait for all parts to be merged
    • Wait for all parts to be deployed to staging
    • Review all parts on staging
    • Wait for all parts to deploy to production
    • Launch the experiment
  • 2022-01-25: Breaking it down into more detail & vertically sliced iterations

    Vertically sliced iterations

    1. ✅ Smallest & simplest MVC possible
      • Add experiment feature flag: cross_stage_fdm
      • Use experiment feature flag to add menu item to the Help (a.k.a. ?) menu
        • Limit the experiment to owners/admins of free namespaces
        • New menu item takes user to our feature discovery moment page
      • Track when a user opens the ? menu (both variants)
        • This is already happening, we just need to add our experiment context to that tracking: experiement=cross_stage_fdm
        • Current tracking is: action=click_question_mark_link, label=main_navigation, property=navigation
      • Track when a user clicks on the new menu item (candidate only)
      • Track render (as an FE event) of feature discovery page
      • Create MVC version of the feature discovery page:
        • Choose an approrpriate layout (or create a new one)
        • Title: “Discover Premium & Ultimate.”
        • Copy: “Access advanced features, build more efficiently, strengthen security and compliance.”
        • Add the “Start a free trial” CTA
          • Add the “Start a free trial” button to the feature discovery page
          • Add the desired glm params to the button: glm_source=gitlab.com&glm_content=cross_stage_fdm
            • Verify that these params make it all the way through to the leads table in CustomersDot
          • Track clicks of the “Start a free trial” CTA button
        • Add the “Contact sales” CTA
          • Add the “Contact sales” CTA button to the feature discovery page
          • Pull in the necessary JS for the hand_raise_lead_button.vue component to work
          • Track clicks of the button
          • Figure out how to get the desired glm params to be submitted with the form: glm_source=gitlab.com&glm_content=cross_stage_fdm
            • Verify that these params make it all the way through to the leads table in CustomersDot
        • Promo image (the one taken from the about.gitlab.com/enterprise page)
      • Add tests for all the new things
        • A “happy path” feature spec (follows the flow of the experiment ensuring the menu item exists, the page renders, and the buttons exist & are clickable)
        • Helper specs (testing new helper)
        • Routing specs (for the new route)
        • View specs (testing the contents of the view)
          • New dashboard view
          • New help menu partial
            • Can only do a “happy path” test. If we try to test the non-rendering cases, we hit one of two issues:
              • Error in ./ee/spec/views/layouts/header/help_dropdown/_cross_stage_fdm.html.haml_spec.rb:34: no implicit conversion of nil into String
              • ee/app/views/layouts/header/help_dropdown/_cross_stage_fdm.html.haml:3 [W] RuboCop: Lint/TopLevelReturnWithArgument: Top level return with argument detected.
        • Request specs (testing new controller & action)
        • Update JS specs for hand_raise_lead_button.vue
    2. ✅ Add the “Speed. Efficiency. Trust.” section
      • Add the title: “Speed. Efficiency. Trust.”
      • Create data objects for the three cards
      • Figure out how to pull in the SVG icons
      • Style it up
      • Update the view specs
    3. ✅ Add the “Access advanced features.” section
      • Add the title: “Access advanced features.”
      • Create data objects for the 10 cards
      • Use the same method for pulling in the SVG icons
      • Style it up
      • Update the view specs
    1. Add the semi-sticky footer
      • Add a footer to the layout (might need to create your own at this point)
        • Make it semi-sticky to the viewport (i.e. if the content is less than the height of the viewport, the foot sticks to the very bottom of the viewport, if the content is greater than the height of the viewport, the footer flows with the content)
      • Add the footer title: “Free 30 days trial”
      • Add the footer copy: “Try everything for free. No credit card required.”
      • Reproduce the CTAs from the header into the footer
        • Double-check that it works well to have multiple versions of the hand_raise_lead_button.vue component/app on the page
      • Update the view specs
      • Update the feature specs to click on CTAs in the header & in the footer to make sure both places work as expected
    2. Add social proof sections
      • Add the copy to the header area: “Trusted by enterprises, loved by developers”
      • Add the chosen logos to the header: hotjar, siemens, wish, & KnowBe4
      • Add the “How they use GitLab.” section
        • Add the title: “How they use GitLab.”
        • Create data objects for the three examples
        • Figure out where/how to get the color versions of the logos (this might be the default, actually)
      • Style it all up
      • Update the view specs
    3. Create special styles for the mobile version of the “Access advanced features.” section
      • We want a single row which scrolls horizontally
      • Test & debug the horizontal scrolling effect in multiple browsers
    4. Consider opening up the experiment to owners/admins of trial namespaces
      • Figure out what to do with the “Start a free trial” CTA
      • Adjust the current restriction to also allow owners/admins of trial namespaces
      • Change the “Start a trial” CTA wording, URL, & tracking if the namespace is on a trial
        • Change wording to something like “Upgrade to Ultimate”
        • Change URL to point to the namespace’s billing page
        • Change tracking label to be something like upgrade
      • Update all associated tests, including adding new tests for this new scenario
    5. Consider opening up the experiment to owners/admins of all non-Ultimate namespaces
      • If we change the “Start a trial” CTA to something like “Upgrade to Ultimate”, then we might as well open up the experiment to owners & admins of all non-Ultimate namespaces
      • Remove the restriction to be an owner/admin of a free or trial namespace
      • Add a restriction to be an owner/admin of a non-Ultimate namespace
      • Reuse the same “Upgrade to Ultimate” (or whatever) alternate CTA for non-free-namespace users
      • Update all associated tests, including adding new tests for the new scenario

  • …

Edited by Dallas Reedy