Experiment Tracking: New user invitation email variant 1
Overview
This is an experiment tracking issue for: https://gitlab.com/gitlab-org/growth/team-tasks/-/issues/182 using the scoped experiment labels.
As well as defining the experiment rollout and cleanup, this issue incorporates the relevant
Feature Flag Roll Out
steps.
Owners
- Team:
group::acquisition
- Most appropriate slack channel to reach out to:
#g_acquisition
- Best individual to reach out to: @nicolasdular
Expectations
### What are we expecting to happen?
What might happen if this goes wrong?
What can we monitor to detect problems with this?
Staging Test
/chatops run feature enable invite_email_experiment
Rollout plan:
See link to experiment issue/epic
- Runtime: 30 days or until we reach statistical significance
- We will roll this out behind a feature flag and expose this to 50% of users to start then ramp it up from there.
/chatops run feature set invite_email_experiment 50
Roll Out Steps
-
Enable on staging -
Test on staging -
Ensure that documentation has been updated -
Enable on GitLab.com for individual groups/projects listed above and verify behavior. -
Announce on the issue an estimated time this will be enabled on GitLab.com -
Enable on GitLab.com by running chatops command in #production
-
Cross post chatops slack command to #support_gitlab-com
(more guidance when this is necessary in the dev docs) and in your team channel -
Announce on the issue that the flag has been enabled -
Remove feature flag and add changelog entry -
If the experiment becomes the default, let's find a way how to add the new footer to the _mailer
template, see gitlab-org/gitlab!39628 (comment 399339547) for more info
-
-
After the flag removal is deployed, clean up the feature flag by running chatops command in #production
channel
Snowplow Events
-
When sending an email:
{ name: Growth::Acquisition::Experiment::InviteEmail, action: 'sent', property: 'control_group' OR 'experiment_group' }
-
When user opens the invite link, we also send a hash that is consistent for each invite. We need to unify on this hash, because the link can be opened multiple times
{ name: Growth::Acquisition::Experiment::InviteEmail, action: 'opened', property: 'control_group' OR 'experiment_group', label: UNIQUE_HASH_PER_INVITE }
-
When user accepts the invitation
{ name: Growth::Acquisition::Experiment::InviteEmail, action: 'accepted', property: 'control_group' OR 'experiment_group', label: UNIQUE_HASH_PER_INVITE }
Results:
Edited by Nicolas Dular