Experiment Rollout: Add user invitation track to day zero for "company" namespaces
Summary
This issue tracks the rollout and status of an experiment through to removal.
- Experiment key / feature flag name:
invite_team_email
- Epic or issue link: #342980 (closed)
This is an experiment rollout issue
using the scoped experiment label.
As well as defining the experiment rollout and cleanup, this issue incorporates the relevant
Feature Flag Roll Out
steps.
Owners
- Team:
group::activation
- Most appropriate slack channel to reach out to:
#g_activation
- Best individual to reach out to: @eugielimpin
- Product manager (PM): @jstava
Stakeholders
Expectations
What are we expecting to happen?
What might happen if this goes wrong?
What can we monitor to detect problems with this?
Tracking
Note: This was copied with modifications from gitlab-org/growth/team-tasks#303 (closed) as this experiment tracks the same data as in-product marketing emails.
Database Tracking
Every namespace that qualifies for sending an invite team email will be placed in the experimental
or the control
group. Emails will only be sent to namespaces in the experimental
group: (the values for experiment_id (9) and namespace_id (999) are fictive)
experiments
table
id | name |
---|---|
9 | invite_team_email |
experiment_subjects
table
experiment_id | namespace_id | variant |
---|---|---|
9 | 999 | {0,1} |
Variant meaning:
0: control group
1: experimental group
The namespace_id
from the experiments_subjects
table can be joined with the namespace_id
from the onboarding_progresses
table in order to determine conversion. For example, counting the number of namespaces for the experimental
variant that added users:
select count(*)
from experiment_subjects subject
inner join experiments on experiments.id = subject.experiment_id
inner join onboarding_progresses progress on subject.namespace_id = progress.namespace_id
where experiments.name = 'invite_team_email'
and subject.variant = 1
and progress.user_added_at is not null
Snowplow Tracking
On top of tracking the onboarding progress for namespaces that took part in the experiment, Snowplow events are sent when users click on the CTA link in the emails. The recorded unstructured self-describing events are of the type email_campaigns
and look like this:
{
"namespace_id": 999,
"track": "invite_team",
"series": 0,
"subject_line": "Invite your teammates to GitLab"
}
Possible values:
track | series |
---|---|
invite_team | 0 |
Dashboard for monitoring CTA clicks across in-product marketing emails and invite team emails
https://app.periscopedata.com/app/gitlab/950037/Alex-Buijs'-Dashboard?widget=13212077&udv=0
Tracked data
Note: you can use the CXL calculator to determine if your experiment has reached significance. The calculator includes an estimate for how much longer an experiment must run for before reaching significance.
Rollout plan
- Runtime in days, or until we expect to reach statistical significance:
30
- We will roll this out behind a feature flag and expose this to 50% of actors to start then ramp it up from there.
/chatops run feature set invite_team_email 50 --actors
Status
Preferred workflow
The issue should be assigned to the Product manager (PM) or Engineer (Eng) as follows:
- PM determines and manages the status of the experiment (assign this issue to the PM)
- PM asks for initial rollout on production, or changes to the status (assign to an Eng)
- Eng changes the status using
chatops
(reassign to the PM) - When concluded, PM updates the 'Roll Out Steps' and adds a milestone (assigns to an Eng)
The current status and history can be viewed using the:
- API (GitLab team members)
- Feature flag log (GitLab team members)
- Experiment rollout board
In this rollout issue, ensure the scoped experiment::
label is kept accurate.
Experiment Results
Roll Out Steps
-
Confirm that QA tests pass with the feature flag enabled (if you're unsure how, contact the relevant stable counterpart in the Quality department) -
Enable on staging ( /chatops run feature set invite_team_email true --staging
) -
Test on staging -
Ensure that documentation has been updated -
Enable on GitLab.com for individual groups/projects listed above and verify behaviour ( /chatops run feature set --project=gitlab-org/gitlab feature_name true
) -
Coordinate a time to enable the flag with the SRE oncall and release managers - In
#production
mention@sre-oncall
and@release-managers
. Once an SRE on call and Release Manager on call confirm, you can proceed with the rollout
- In
-
Announce on the issue an estimated time this will be enabled on GitLab.com -
Enable on GitLab.com by running chatops command in #production
(/chatops run feature set feature_name true
) -
Cross post chatops Slack command to #support_gitlab-com
(more guidance when this is necessary in the dev docs) and in your team channel -
Announce on the issue that the flag has been enabled -
Remove experiment code and feature flag and add changelog entry - a separate cleanup issue might be required -
After the flag removal is deployed, clean up the feature flag by running chatops command in #production
channel -
Assign to the product manager to update the knowledge base (if applicable)
Rollback Steps
-
This feature can be disabled by running the following Chatops command:
/chatops run feature set invite_team_email false
Experiment Successful Cleanup Concerns
Clean up issue: #355706 (closed)