Hypothesis: New onboarding will improve return rates among completers

Overview

GitLab currently does not have a strong onboarding experience that guides new users to an "ah-ha" moment that drives stickiness. The current experience after joining simply lets them explore the application, but GitLab is a broad application and showing a user some of GitLab's capabilities may help provide them with a better understanding of why the product is valuable.

Since new users are early in any type of engagement funnel, helping these users find value and stick around in the product has a disproportionate amount of value in helping drive retention.

What we hope to find

• Statistically significant proof to reject the null hypothesis that new onboarding does not improve user return rates. We'll test this by measuring:

  • Week 1 - 4 retention (among new users, how many returned in the following weeks).

• Method: conduct an A/B test to run for ~4 weeks. We hope to see a statistically significant lift in our retention curve in the test variant vs. control variant.

  • Variant 1: Control (no onboarding). % in variant TBD.
  • Variant 2: New onboarding. % in variant TBD

Plan

• Design an improved onboarding experience in %11.11.
• Ship new onboarding in %12.1 behind a feature flag.
• QA onboarding: functionality, analytics, and A/B test.
• Once onboarding is available on GitLab.com in an RC, enable this flag for a % of our userbase. (early July)
• Check the experiment data 2 days after enabling the flag to validate that we're getting measurable results.
• Read-out the experiment results in early June.
• Fix future releases to the winning variant.