[Experiment Rollout] signup_intent_step_one
Summary
This issue tracks the rollout and status of an experiment through to removal.
- Feature flag name:
signup_intent_step_one
- Epic or issue link: epic
This is an experiment rollout issue
using the scoped experiment label.
As well as defining the experiment rollout and cleanup, this issue incorporates the relevant
Feature Flag Roll Out
steps.
Owners
- Team:
group::acquisition
- Most appropriate slack channel to reach out to:
#g_acquisition
- Best individual to reach out to: @robyrne
- Product manager (PM): @p_cordero
Stakeholders
Expectations
What are we expecting to happen?
What might happen if this goes wrong?
QA
-
PM (@p_cordero) to tick this off when all QA has been completed and passed
Confirm experiment on STAGING
- Go to
https://staging.gitlab.com/-/trial_registrations/new
and view step 1 of trial registration
Assignment
Confirm that assignment events are populating on STAGING https://10az.online.tableau.com/#/site/gitlab/views/DRAFTPDExperimentEventValidation/GrowthExperimentEventValidationDashboard?:iid=1
Control / Candidate experience
Does the control & candidate user experience work as expected? Does it match designs? Discover potential edge cases?
-
PM (@p_cordero) to confirm this
Control
-
Confirm that Control experience works as expected #441494 (closed) -
User does not see any signup intent dropdown -
User is able to complete step 1 of registration (may as well test the complete registration path as well)
-
Candidate
-
Confirm that Candidate experience works as expected #441494 (closed) -
User sees the signup intent dropdown -
User is able to complete step 1 of registration (may as well test the complete registration path as well) for each selection -
set up a new team -
set up a new personal account -
join an existing team -
contribute to a public project on GitLab
-
-
Confirm that a user must select an intent before proceeding
-
What can we monitor to detect problems with this?
Tracked data
Tracking
-
PM (@p_cordero) confirm that all events are populating on STAGING on https://10az.online.tableau.com/#/site/gitlab/views/DRAFTPDExperimentEventValidation/GrowthExperimentEventValidationDashboard?:iid=1 -
When filtering for dates, ensure to select
2024-03-22
-
For analysis, please filter to
2024-03-25
as the start date for production performance analysis
Tracked data
Note: you can use the CXL calculator to determine if your experiment has reached significance. The calculator includes an estimate for how much longer an experiment must run for before reaching significance.
Rollout plan
- Runtime in days, or until we expect to reach statistical significance:
30
- We will roll this out behind a feature flag and expose this to
<rollout-percentage>
% of actors to start then ramp it up from there.
/chatops run feature set <feature-flag-name> <rollout-percentage> --actors
Status
Preferred workflow
The issue should be assigned to the Product manager (PM) or Engineer (Eng) as follows:
- PM determines and manages the status of the experiment (assign this issue to the PM)
- PM asks for initial rollout on production, or changes to the status (assign to an Eng)
- Eng changes the status using
chatops
(reassign to the PM) - When concluded, PM updates the 'Roll Out Steps' and adds a milestone (assigns to an Eng)
The current status and history can be viewed using the:
- API (GitLab team members)
- Feature flag log (GitLab team members)
- Experiment rollout board
In this rollout issue, ensure the scoped experiment::
label is kept accurate.
Experiment Results
Roll Out Steps
-
Confirm that end-to-end tests pass with the feature flag enabled. If there are failing tests, contact the relevant stable counterpart in the Quality department to collaborate in updating the tests or confirming that the failing tests are not caused by the changes behind the enabled feature flag. - See
#qa-staging
Slack channel and look for the following messages:- test kicked off:
Feature flag <feature-flag-name> has been set to true on **gstg**
- test result:
This pipeline was triggered due to toggling of <feature-flag-name> feature flag
- test kicked off:
- See
-
Enable on staging ( /chatops run feature set <feature-flag-name> true --staging
) -
Test on staging -
Ensure that documentation has been updated -
Enable on GitLab.com for individual groups/projects listed above and verify behaviour ( /chatops run feature set --project=gitlab-org/gitlab <feature-flag-name> true
) -
Coordinate a time to enable the flag with the SRE oncall and release managers - In
#production
mention@sre-oncall
and@release-managers
. Once an SRE on call and Release Manager on call confirm, you can proceed with the rollout
- In
-
Announce on the issue an estimated time this will be enabled on GitLab.com -
Enable on GitLab.com by running chatops command in #production
(/chatops run feature set <feature-flag-name> true
) -
Cross post chatops Slack command to #support_gitlab-com
(more guidance when this is necessary in the dev docs) and in your team channel -
Announce on the issue that the flag has been enabled -
Remove experiment code and feature flag and add changelog entry - a separate cleanup issue might be required -
After the flag removal is deployed, clean up the feature flag by running chatops command in #production
channel -
Assign to the product manager to update the knowledge base (if applicable)
Rollback Steps
-
This feature can be disabled by running the following Chatops command:
/chatops run feature set <feature-flag-name> false
Experiment Successful Cleanup Concerns
Items to be considered if candidate experience is to become a permanent part of GitLab