Experiment Tracking: `customize_homepage` banner
Overview
This is an experiment tracking issue for: https://gitlab.com/gitlab-org/gitlab/-/issues/225858 using the scoped experiment label.
As well as defining the experiment rollout and cleanup, this issue incorporates the relevant
Feature Flag Roll Out
steps.
Owners
- Team: ~"group::expansion"
- Most appropriate slack channel to reach out to:
#g_expansion
- Best individual to reach out to: @timhey
Expectations
What are we expecting to happen?
We are expecting to show a banner at the top of the home page on gitlab.com. This will call attention to a setting the many of our users have left to its default setting. We are expecting to learn whether or not users are interested in making a change to this setting.
What might happen if this goes wrong?
I believe that if something were to go wrong we there would be little to no impact on our users or application. The worst-case scenario is that users speak up and share feedback in a negative fashion which we would respond to swiftly by toggling off the feature flag.
What can we monitor to detect problems with this?
We will create a dashboard to monitor and track this experiment.
Staging Test
To force this experiment on staging use ?force_experiment=customize_homepage
See #218 (closed)
Rollout plan:
See https://gitlab.com/gitlab-org/gitlab/-/issues/225858
- Runtime: 30 days or until we reach statistical significance
- We will roll this out behind a feature flag and expose this to 20% of users on GitLab.com to start then ramp it up from there.
- Experiment key
customize_homepage
(seeexperimentation.rb
in gitlab, append '_experiment_percentage')
/chatops run feature set customize_homepage_experiment_percentage <INITIAL_PERCENTAGE>
-
2020-12-04 Re-enabled at 20% in production -
2020-12-04 increase to 100% in production -
Review and schedule cleanup https://gitlab.com/gitlab-org/gitlab/-/issues/299895
Beta groups/projects
- ...
- ...
Roll Out Steps
-
Enable on staging -
Test on staging -
Ensure that documentation has been updated -
Enable on GitLab.com for individual groups/projects listed above and verify behavior. -
Announce on the issue an estimated time this will be enabled on GitLab.com -
Enable on GitLab.com by running chatops command in #production
-
Cross post chatops slack command to #support_gitlab-com
(more guidance when this is necessary in the dev docs) and in your team channel -
Announce on the issue that the flag has been enabled -
Remove feature flag and add changelog entry -
After the flag removal is deployed, clean up the feature flag by running chatops command in #production
channel
Results:
Dashboard: https://app.periscopedata.com/app/gitlab/807895/Growth-Expansion-Experiment:-customize-home-page-banner (request https://gitlab.com/gitlab-data/analytics/-/issues/5901).