[WIP] Standardize Experimentation Framework
Update: the Experimentation working group &46 (closed) is discussing experimentation - initially for GitLab.com, but consideration for self managed. At this stage other GitLab products are out of scope (about, customers), although build vs buy will be covered.
Overview
We currently have a rudimentary experimentation framework with several limitations:
- Cross-session experimentation does not work. i.e. if you use your phone browser vs your laptop browser you may be put into different groups
- On GitLab.com, increasing % rollout group size requires an MR. i.e. bumping an experiment from 10% rollout to 50% rollout requires an MR and deployment.
- It's difficult to track experiments between logged in vs logged out states
- It's difficult to track experiments across domains
The goal of this issue is to work towards standardizing our experimentation framework across all of GitLab's domains and services to allow us to rapidly experiment across the full funnel.
Standardize Experimentation Tools
We currently use three different tools for experimentation:
- Flipper (used on gitlab in experiment.rb)
- Unleash (used on customers)
- LaunchDarkly (used on about.gitlab.com by marketing)
Enable Cross Domain Experimentation (Cross Domain Tracking)
- gitlab.com
- customers.gitlab.com
- about.gitlab.com
Self-Hosted Experiments
- Explore if this makes sense and how this would be possible gitlab-org/gitlab#209098
Tracking Active/Previous Experiments
- We should have a centralized place in the product growth handbook to track experiment issues and their results #105 (closed)
Opting Out of Experiments
-
We should allow users to opt out of experiments https://gitlab.com/gitlab-org/gitlab/issues/119107 -
We should respect a browser's DNT setting gitlab-org/gitlab!22100 (merged)
Next Steps
-
On GitLab.com: Adjust experimentation group without MRs gitlab-org/gitlab#200015 (closed) -
...
Edited by Phil Calder