[WIP] Standardize Experimentation Framework
Moved to gitlab-org/growth/team-tasks#106 (closed)
Overview
We currently have a rudimentary experimentation framework with several limitations:
- Cross-session experimentation does not work. i.e. if you use your phone browser vs your laptop browser you may be put into different groups
- On GitLab.com, increasing % rollout group size requires an MR. i.e. bumping an experiment from 10% rollout to 50% rollout requires an MR and deployment.
- It's difficult to track experiments between logged in vs logged out states
- It's difficult to track experiments across domains
The goal of this issue is to work towards standardizing our experimentation framework across all of GitLab's domains and services to allow us to rapidly experiment across the full funnel.
Standardize Experimentation Frameworks
- Flipper (used on gitlab in experiment.rb)
- Unleash (used on customers)
- LaunchDarkly (used on about.gitlab.com by marketing)
Enable Cross Domain Experimentation (Cross Domain Tracking)
- gitlab.com
- customers.gitlab.com
- about.gitlab.com
Self-Hosted Experiments
- Explore if this makes sense and how this would be possible #209098
Tracking Active/Previous Experiments
- We should have a centralized place in the product growth handbook to track experiment issues and their results gitlab-org/growth/team-tasks#105 (closed)
Opting Out of Experiments
- We should allow users to opt out of experiments https://gitlab.com/gitlab-org/gitlab/issues/119107
- We should respect a browser's DNT setting !22100 (merged)
Next Steps
-
On GitLab.com: Adjust experimentation group without MRs #200015 (closed) -
...
Edited by Jerome Z Ng