Activation planning for 12.2
Overview and objectives
12.2 is the first release with a set of dedicated folks to help advance our Growth goals. Exciting! Here's my take on our current objectives:
Where we're at today
Today, we're still operating in a relatively data-blind environment. We don't have a strong ability to be able to attach outcomes to many of the things that we ship. We'd like to iterate toward being able to ship and quickly learn from experiments, then start aggressively shipping ways of improving some of the core metrics associated with our respective goals. Specifically, what we need:
- An improved way of measuring the impact of our experiments,
- An experimentation framework that allows us to analyze the effects of our tests and lastly,
- Ship and validate ideas against the goals we've set for the Adoption group.
Let's discuss each:
Measuring impact
To further Activation's specific goals, we need to be able to measure our defined activation goals on both GitLab.com and self-managed.
- We have an issue solving this for GitLab.com.
- We need to be able to measure this on self-managed.
We also have broader needs at GitLab, Inc. to measure stage activity (e.g. for each individual stage, what level of usage are we seeing in SMAU?).
A new development is the incoming introduction of Pendo (implementation timeline TBD, likely ~1 month).
- Please see the summary from the Pendo test plan we executed, if interested.
Pendo solves some of these immediate needs by providing us with a strong client-side framework for capturing Activation metrics (on GitLab.com, at minimum). So, what problems will still exist after Pendo's introduction?
- Pendo may prove to be a poor solution for self-managed customers, since it'll send a lot of outbound traffic to Pendo.io that might be unpalatable for these customers.
- Pendo does not track backend events that aren't easily traced to a user.
Therefore, we'll have a need to continue to iterate on a data collection framework outside of Pendo. However, Pendo likely solves for the immediate urgency to continue working on usage ping improvements. We'll need to continue work on usage ping and eventually use Snowplow to track usage, but we can de-prioritize these improvements today.
Experimentation framework
Pendo does not solve for our A/B testing needs (we'd like to A/B test most of our experiments to understand the impact on our users and to validate they're hitting on our goals). We should invest in finding a longer-term solution for the flipper A/B testing framework and consider how this can intersect with Feature Flags in GitLab:
Improve Activation
If we're able to successfully A/B test and measure the impact of our tests, we're in great shape and we're ready to start shipping tests behind flags for rollout.
New onboarding
New onboarding has been developed behind a flag. During testing (thanks Jacki and Katherine), we discovered a number of items that should be addressed:
- See https://gitlab.com/gitlab-org/growth/issues/71#note_194548556. Issues to be created.
- Skip onboarding on mobile.
Once we're ready, we should roll onboarding out to GitLab.com and measure the impact.
Soft email confirmation
We can also start working on other tests. We could build and ship soft email confirmation behind a feature flag.
Log user in immediately on email confirmation
Some additions we don't really need to A/B test; if they're straightforward improvements, we should just do them and not commit to the added weight of A/B testing and analyzing the results. Logging the user in on email confirmation is one of them.
reCAPTCHA alternative
We discovered that reCAPTCHA is harming registration rates significantly. We should prioritize finding a solution that is user friendly, but also helps manage registration spam.
Note that Plan was tackling this in https://gitlab.com/gitlab-org/gitlab-ce/issues/45684#note_187540189, but that issue seems to have been stalled.
Other ideas
Please contribute other ideas on improving Adoption metrics by creating issues in the Growth project.
Board
For specific issues and prioritization, please see https://gitlab.com/groups/gitlab-org/-/boards/1158847.