MVC: GitLab A/B Testing
Release notes
A/B Testing is one of the most practical Progressive Delivery methods to build an effective product by collecting analytics data. Sometimes, when you ship a feature, you don't have enough confidence that which pattern/variation/feature-spec is the best fit for end-users. A/B testing allows you to ship a feature quickly with measuring the impact of it and eventually contribute to the factor of expanding user base of your application.
Problem to solve
In a typical DevOps lifecycle, often Product Manger and UX designer don't have enough time to research what's the most effective specification on a feature design.
Intended users
User experience goal
- Users can collect application analytics data from Feature Flag client.
- Users can define experiments to visualize the collected data.
Proposal
Here are the MVC steps
- Step 1. Introduce Event API to collect analytics data from Unleash Client.
- Step 2. Visualize the event data as an experiment.
Step 1. Introduce Event API to collect analytics data from Unleash Client
- We introduce a new Event API in Unleash. This API allows Unleash client to send analytics data to Unleash server.
- The data schema is the following:
events:
- recorded_at: # When the event happened e.g. `2020-11-02T13:15:34.787Z`
name: # What event happened e.g. `user-visits-new-project-page`
context: # Who made the event e.g. `jhon`
type: # The type of event. e.g. `engage`, `variant`
output: # The output of event. If it's `varitnat` type, the variant value. e.g. `blue`
- We persist the data into
environment_events
table. - Important: We don't have a consensus with Unleash Official about this API design yet. Likely, we need to contribute to their server at first.
- Here is an example of Event API extension on unleash-ruby-client
Step 2. Visualize the event data as an experiment
- As we've started collecting analytics data in GitLab, we can visualize it in various ways.
- We create
A/B Testing
index page.
# Database table in GitLab
operations_experiments
- environment_id: FK
- name: The name of the experiment
- base_event_name: The experimental event
- base_event_type: The experimental event
- base_event_output: The experimental event
- convertion_event_name: The goal of the event
- convertion_event_type: The goal of the event
- convertion_event_output: The goal of the event
#280536 (comment 448058957))
Not in MVC (based on- Step 3. Support Unleash Variant in GitLab Feature Flag
Unleash Variant in GitLab Feature Flag
Step 3. Support- Unleash supports Unleash Variant to allow a feature flag to return a variant instead of traditional boolean value.
- We extends GitLab Feature Flag feature to allow to define variants along side with strategies.
# Database table in GitLab
operations_feature_flag_variants
- feature_flag_id: # FK
- name: # The name of the variant e.g. `blue`
- weight: # The weight of the variant e.g. `50`
- payload: # The payload of the variant (optional)
- overrides: # The rules overrides the `weight` (optional)
PoC
Here is the PoC MR.
You can see the demo in https://youtu.be/vzE0aiv7zoc?t=1145
Further details
Permissions and Security
Documentation
Availability & Testing
What does success look like, and how can we measure that?
What is the type of buyer?
Is this a cross-stage feature?
Links / references
This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.