Experiment guide for Product Managers
Validate your experiment idea:
Follow the experimentation process for cultivating your experimentation backlog.
Validate your success metric:
What action will determine the success of this experiment? For example, is it a CTA click, is it adoption of a particular feature? Is it upgrading or starting a trial?
Tracking the user experience:
Based on your success metric are there any additional data points you need to monitor from users in the control vs experiment?
Adding new tracking events into existing data tables:
Creating net new data tables and events:
Confirm table joins exist to support your events and success metric:
Once your success metric and events are defined you should ensure that the existing tables in Sisense exist to support any query you write to track the success of your experiment. For example, if we wanted to track if net new signups that went through particular onboarding were more likely to adopt a particular stage (a backend tracking based experiment). We'd need to confirm in Sisense that we can join the tables of when a net-new user is created (user_id), to our experiments enrollment table (control vs experiment) where we record 'user_id' to a table that shows users (user_id) who created namespaces along with the namespace_ID of the namespace they created and finally we could join this to our SMAU tables to understand stage adoption rate by particular `namespace_IDs.
If you find any gaps in the joins that you require, for example, if you were unable to find the ID of the user that created the namespace in the example above you should follow up with the Product Analyst team prior to launching your experiment to ensure it's possible to track your events and success metric.
Deliverable
Updated guide for Product Managers at GitLab on how to propose an idea for an experiment
See also
https://about.gitlab.com/handbook/engineering/development/growth/#experimentation-process