Discuss Assignment Requirements for Experiments
Summary
To ensure experiments are implemented in such a way that we are able to analyze our hypothesis, Product Managers specify certain Assignment requirements that dictate who and when someone is assigned into an experiment. These requirements are implemented by engineering or in some cases, filtered out in analysis post-experiment.
Some examples:
- Free or Trial or both
- New or existing or both
- There have been examples of more specific requirements. This most recently came up in the implementation of the empty state experiment
The purpose of this issue is to capture discussion of Assignment requirements
@s_awezec @eneuberger @gdoud @kniechajewicz @jhalloran @gitlab-org/growth/engineers
Context
In the implementation issue for the empty state experiment, PM raised a concern regarding the data we will get back if we do not add additional assignment requirements
Is it possible to only initiate 'assignment' for new free/trial signups created after
2024-04-17
when they render the Issues/MRs empty state page. My interests are with wanting to ensure the audience we analyze in this experiment actually 'gets the chance' to see empty state experience we are testing.
If we just do it upon sign in, we can't control the number in each group of who sees the empty state page and it's possible one group (Control or Candidate) happens to have more in it who don't ever see the empty state page and that influences their activity.
Imagine a clinical study where we are testing the impact of a certain medicine. One group is given the real medicine and another is given a placebo. In order for the data/analysis to be accurate, all people in the group either have to take the real medicine or the placebo.
I am worried that there will be folks in either of our candidate or control group that will not 'view' the empty state page and that will impact the data we get back in the experiment.
I think this kind of specific 'assignment' requirement could become more and more common for experiments as we get deeper into the Product (user will have to navigate somewhere to see the experiment) with Activation as well as Conversion. Previous experiments have not had to worry about this because the 'experiment' was offered to everyone (they went through the registration flow, they landed on Learn GitLab by default).
Conclusion from discussion
- We have added clarity to how assignment is currently handled and it satisfies the original concern that assignments were occurring without experiencing the experiment #455632 (comment 1857707855)
- We have a path forward to capturing requirements for experiments more efficiently (Assignment, Tracking, UX). If there are cases where assignment requirements are not currently possible this should be realized during workflowsolution validation thread of Assignment #455632 (comment 1858328748)