Experiment Tracking: ci_notification_dot
Overview
GitLab.com notification dot when 'buy ci minutes' button is displayed in user settings menu.
This is an experiment tracking issue for: &30 using the scoped experiment labels
- Experiment key:
ci_notification_dot
- Feature flag:
ci_notification_dot_experiment_percentage
Note that &28 and tracking issue https://gitlab.com/gitlab-org/growth/team-tasks/-/issues/102 takes precedence over this experiment.
gitlab-org/gitlab!27626 (merged)
# https://gitlab.com/gitlab-org/gitlab/-/commit/cef0c04cef1bdbe931e0416ad5aa13f7cee6baed
ci_notification_dot: {
tracking_category: 'Growth::Expansion::Experiment::CiNotificationDot'
}
Staging Test
To force this experiment on staging use ?force_experiment=ci_notification_dot
See #120 (closed)
Rollout plan:
- Start by exposing this to 20% of applicable users and ramp it up as we go.
- Increase the % of users who see this every 2 weeks if the results are positive.
- Runtime: 30 days or until we reach statistical significance
- Eventually either rollout to 100% or disable the feature flag.
/chatops run feature set ci_notification_dot_experiment_percentage 10
Action dates
-
2020-08-10 - Enabled feature flag and exposed to 20% of users on gitlab.com -
2020-11-30 - Let's enable the feature flag and expose to 100% of users on gitlab.com -
2021-01-14 - PM declared this experiment a success #101 (comment 485882384) - adding Experiment Cleanup issue for discussion https://gitlab.com/gitlab-org/gitlab/-/issues/298795
See the due date for next review
Beta groups/projects
-
https://gitlab.com/testing-ci-minutes-balance
group - ...
Tracking
Activity being tracked | track_event | track_label | track_property |
---|---|---|---|
User avatar contains notification dot | render |
show_buy_ci_minutes_notification |
plan name |
User Nav dropdown opened and contains Buy CI minutes link |
show_buy_ci_minutes |
plan name | user_dropdown |
Buy CI minutes link clicked from User Nav dropdown |
click_buy_ci_minutes |
plan name | user_dropdown |
Buy additional minutes clicked from quota page |
click_buy_ci_minutes |
plan name | pipeline_quota_page |
Roll Out Steps
-
Enable on staging -
Test on staging #120 (closed) -
Ensure that documentation has been updated -
Enable on GitLab.com for individual groups/projects listed above and verify behavior -
Coordinate a time to enable the flag with #production
and#g_delivery
on slack. -
Announce on the issue an estimated time this will be enabled on GitLab.com -
Enable on GitLab.com by running chatops command in #production
-
Cross post chatops slack command to #support_gitlab-com
(more guidance when this is necessary in the dev docs) and in your team channel -
Announce on the issue that the flag has been enabled -
Remove feature flag and add changelog entry -
After the flag removal is deployed, clean up the feature flag by running chatops command in #production
channel
Results:
Experiment Clean Up
Follow up issue to clean up this experiment including removal of feature flag https://gitlab.com/gitlab-org/gitlab/-/issues/298795
Edited by Phil Calder