Add test/control group indicators to the database
This issue is related to: Onboarding through issue board
Problem
Due to user_id tracking limitations, we cannot track users through their lifecycle and usage behaviors between different features together. Instead, we have to use aggregate metrics in order to understand the impacts of an experiment overall. Because of this, it is extremely important to be able to segment users by the experiment group that they were or are a part of so that we can aggregate usage metrics by that test group.
As an example, given this experiment, we are trying to determine if the new onboarding flow leads to increased success in users getting started with GitLab. We chose to track the experiment group passively by counting users in the experiment if they responded to a question only asked in the experiment group, if the value of that response is null
they are considered to be in the test group.
Data Reference:
- user_preferences.experience_level
-
0
indicates anovice
value -
1
indicates aexperienced
value -
null
indicates user went through the control variant
-
The issue with this methodology is that there are users entering the platform via invitations or other means that also have a null value for this field. Additionally, users that signed up before the experiment was live have a null value for this field. When completing an analysis, it is impossible to segment users who were a part of this experiment and view their aggregate behavior. Moreover, we cannot use frontend events as snowplow does not fire frontend events for users in countries regulated by GDPR (e.g. Germany).
Proposal
EXPLICITLY track users and namespaces as participating in an experiment in the database so that we can join those users with the aggregate metrics used to evaluate a given experiment. At a minimum, we would need the following:
user_id | namespace_id | test_name | test_group |
---|---|---|---|
NOT NULL | OPTIONAL (included only if we know the namespace at the start of an experiment) | NOT NULL | NOT NULL (0 if user is in the control, 1++ if the user is a part of a test variant. E.g. 2 if the user is in variant 2 of test A) |
This is needed in order to relaunch and track Onboarding through issue board