Add Feature Flags A/B testing support
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
Problem to solve
A/B testing is a way to compare two versions of a single variable, typically by testing a subject's response to variant A against variant B, and determining which of the two variants is more effective. We already have support for feature flag percent rollout but we should expand on it so that users can experiment more and roll out the best features that they can.
Intended users
- Release managers
- Support managers
- Customer success managers (for beta/ test users)
- Developers awaiting feedback to select the "right" feature path
Internal groups:
- Marketing
- Growth
Further details
You can now extend feature toggles with multiple variants. This feature enables you to extend a feature toggle to divide your traffic among a set of variants.
Support for the following languages
- node
- java
- ruby
- python
Proposal
We will build A/B testing on top of Unleash's variant solution. Unleash has a simple way to divide the traffic equally.
N-represents the number of options, each variant will be routed to 100/N%
For example: for 5 variants each variant will be 20% of the traffic (as seen in the screenshot above)
- We will allow users to customize the percentage and not bind them to equal percentages.
- The default percentage shall be equal, but it can be changed.
- (Sum of all weights cannot exceed 100), if a user configures that an error message saying "Weights cannot be larger than 100" should be displayed
A/B testing should not be thought about as a stand-alone strategy. It comes paired to UserID or percent rollout.
UX TBD
Permissions and Security
Documentation
Availability & Testing
What does success look like, and how can we measure that?
Number of users using a/b testing strategy
What is the type of buyer?
Is this a cross-stage feature?
Links / references
This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.